Welcome to the February 2022 report from the Reproducible Builds project. In these reports, we try to round-up the important things we and others have been up to over the past month. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.

---

Jiawen Xiong, Yong Shi, Boyuan Chen, Filipe R. Cogo and Zhen Ming Jiang have published a new paper titled Towards Build Verifiability for Java-based Systems (PDF). The abstract of the paper contains the following:

Various efforts towards build verifiability have been made to C/C++-based systems, yet the techniques for Java-based systems are not systematic and are often specific to a particular build tool (eg. Maven). In this study, we present a systematic approach towards build verifiability on Java-based systems.

---

GitBOM is a flexible scheme to track the source code used to generate build artifacts via Git-like unique identifiers. Although the project has been active for a while, the community around GitBOM has now started running weekly community meetings.

---

The paper Chris Lamb and Stefano Zacchiroli is now available in the March/April 2022 issue of IEEE Software. Titled Reproducible Builds: Increasing the Integrity of Software Supply Chains (PDF), the abstract of the paper contains the following:

We first define the problem, and then provide insight into the challenges of making real-world software build in a reproducible manner-this is, when every build generates bit-for-bit identical results. Through the experience of the Reproducible Builds project making the Debian Linux distribution reproducible, we also describe the affinity between reproducibility and quality assurance (QA).

---

In openSUSE, Bernhard M. Wiedemann posted his monthly reproducible builds status report.

---

On our mailing list this month, Thomas Schmitt started a thread around the SOURCE_DATE_EPOCH specification related to formats that cannot help embedding potentially timezone-specific timestamp. (Full thread index.)

---

The Yocto Project is pleased to report that it s core metadata (OpenEmbedded-Core) is now reproducible for all recipes (100% coverage) after issues with newer languages such as Golang were resolved. This was announced in their recent Year in Review publication. It is of particular interest for security updates so that systems can have specific components updated but reducing the risk of other unintended changes and making the sections of the system changing very clear for audit. The project is now also making heavy use of equivalence of build output to determine whether further items in builds need to be rebuilt or whether cached previously built items can be used. As mentioned in the article above, there are now public servers sharing this equivalence information. Reproducibility is key in making this possible and effective to reduce build times/costs/resource usage.

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions 203, 204, 205 and 206 to Debian unstable, as well as made the following changes to the code itself:

• Bug fixes:
  • Fix a file(1)-related regression where Debian .changes files that contained non-ASCII text were not identified as such, therefore resulting in seemingly arbitrary packages not actually comparing the nested files themselves. The non-ASCII parts were typically in the Maintainer or in the changelog text. [ ][ ]
  • Fix a regression when comparing directories against non-directories. [ ][ ]
  • If we fail to scan using binwalk, return False from BinwalkFile.recognizes. [ ]
  • If we fail to import binwalk, don t report that we are missing the Python rpm module! [ ]
• Testsuite improvements:
  • Add a test for recent file(1) issue regarding .changes files. [ ]
  • Use our assert_diff utility where we can within the test_directory.py set of tests. [ ]
  • Don t run our binwalk-related tests as root or fakeroot. The latest version of binwalk has some new security protection against this. [ ]
• Codebase improvements:
  • Drop the _PATH suffix from module-level globals that are not paths. [ ]
  • Tidy some control flow in Difference._reverse_self. [ ]
  • Don t print a warning to the console regarding NT_GNU_BUILD_ID changes. [ ]

In addition, Mattia Rizzolo updated the Debian packaging to ensure that diffoscope and diffoscope-minimal packages have the same version. [ ]

Debian-related updates Vagrant Cascadian wrote to the debian-devel mailing list after noticing that the binutils source package contained unreproducible logs in one of its binary packages. Vagrant expanded the discussion to one about all kinds of build metadata in packages and outlines a number of potential solutions that support reproducible builds and arbitrary metadata. Vagrant also started a discussion on debian-devel after identifying a large number of packages that embed build paths via RPATH when building with CMake, including a list of packages (grouped by Debian maintainer) affected by this issue. Maintainers were requested to check whether their package still builds correctly when passing the -DCMAKE_BUILD_RPATH_USE_ORIGIN=ON directive. On our mailing list this month, kpcyrd announced the release of rebuilderd-debian-buildinfo-crawler a tool to parse the Packages.xz Debian package index file, attempts to discover the right .buildinfo file from buildinfos.debian.net and outputs it in a format that can be understood by rebuilderd. The tool, which is available on GitHub, solves a problem regarding correlating Debian version numbers with their builds. bauen1 provided two patches for debian-cd, the software used to make Debian installer images. This involved passing --invariant and -i deb00001 to mkfs.msdos(8) and avoided embedding timestamps into the gzipped Packages and Translations files. After some discussion, the patches in question were merged and will be included in debian-cd version 3.1.36. Roland Clobus wrote another in-depth status update about status of live Debian images, summarising the current situation that all major desktops build reproducibly with bullseye, bookworm and sid . The python3.10 package was uploaded to Debian by doko, fixing an issue where [.pyc files were not reproducible because the elements in frozenset data structures were not ordered reproducibly. This meant that to creating a bit-for-bit reproducible Debian chroot which included .pyc files was not reproducible. As of writing, the only remaining unreproducible parts of a standard chroot is man-db, but Guillem Jover has a patch for update-alternatives which will likely be part of the next release of dpkg. Elsewhere in Debian, 139 reviews of Debian packages were added, 29 were updated and 17 were removed this month adding to our knowledge about identified issues. A large number of issue types have been updated too, including the addition of captures_kernel_variant, erlang_escript_file, captures_build_path_in_r_rdb_rds_databases, captures_build_path_in_vo_files_generated_by_coq and build_path_in_vo_files_generated_by_coq.

Website updates There were quite a few changes to the Reproducible Builds website and documentation this month as well, including:

• Chris Lamb:
  • Considerably rework the Who is involved? page. [ ][ ]
  • Move the contributors.sh Bash/shell script into a Python script. [ ][ ][ ]
• Daniel Shahaf:
  • Try a different Markdown footnote content syntax to work around a rendering issue. [ ][ ][ ]
• Holger Levsen:
  • Make a huge number of changes to the Who is involved? page, including pre-populating a large number of contributors who cannot be identified from the metadata of the website itself. [ ][ ][ ][ ][ ]
  • Improve linking to sponsors in sidebar navigation. [ ]
  • drop sponsors paragraph as the navigation is clearer now. [ ]
  • Add Mullvad VPN as a bronze-level sponsor . [ ][ ]
• Vagrant Cascadian:
  • Remove a stray parenthesis from the Who is involved? page. [ ]

Upstream patches The Reproducible Builds project attempts to fix as many currently-unreproducible packages as possible. February s patches included the following:

• Bernhard M. Wiedemann:
  • btop (sort-related issue)
  • complexity (date)
  • giac (update the version with upstreamed date patch)
  • htcondor (use CMake timestamp)
  • libint (readdir system call related)
  • libnet (date-related issue)
  • librime-lua (sort filesystem ordering)
  • linux_logo (sort-related issue)
  • micro-editor (date-related issue)
  • openvas-smb (date-related issue)
  • ovmf (sort-related issue)
  • paperjam (date-related issue)
  • python-PyQRCode (date-related issue)
  • quimb (single-CPU build failure)
  • radare2 (Meson date/time-related issue)
  • radare2 (Rework SOURCE_DATE_EPOCH usage to be portable)
  • siproxd (date, with Sebastian Kemper + follow-up
  • xonsh (Address Space Layout Randomisation-related issue)
  • xsnow (date & tar(1)-related issue)
  • zip (toolchain issue related to filesystem ordering)
• Chris Lamb:
  • #1005029 filed against ltsp (forwarded upstream).
  • #1005197 filed against pcmemtest.
  • #1005825 filed against hatchling.
  • #1005826 filed against mpl-sphinx-theme (forwarded upstream)
  • #1005827 filed against gap-hapcryst.
  • #1005901 filed against tree-puzzle.
  • #1005954 filed against jcabi-aspects.
  • #1005955 filed against paper-icon-theme.
• Roland Clobus:
  • #1006358 filed against libxmlb.
• Vagrant Cascadian:
  • #1005408 filed against wcwidth.
  • #1005420 filed against xir.
  • #1005421 filed against xir.
  • #1005726 filed against ruby-github-markup.
  • #1005727 filed against ruby-tioga.
  • #1005792 filed against btop.
  • #1005793 filed against libadwaita-1.
  • #1005794 filed against snibbetracker.
  • #1006252 filed against cctbx.
  • #1006254 filed against mdnsd.
  • #1006256 filed against gmerlin.
  • #1006302 filed against beav.
  • #1006385 filed against krita.
  • #1006407 filed against qt6-base.
  • #1006455 filed against onevpl-intel-gpu.
  • #1006471 filed against ruby3.0.
  • #1006473 filed against nix.
  • #1006474 filed against foma.
  • #1006476 filed against ruby3.0.

Testing framework The Reproducible Builds project runs a significant testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:

• Daniel Golle:
  • Update the OpenWrt configuration to not depend on the host LLVM, adding lines to the .config seed to build LLVM for eBPF from source. [ ]
  • Preserve more OpenWrt-related build artifacts. [ ]
• Holger Levsen:
• Temporary use a different Git tree when building OpenWrt as our tests had been broken since September 2020. This was reverted after the patch in question was accepted by Paul Spooren into the canonical openwrt.git repository the next day.
  • Various improvements to debugging OpenWrt reproducibility. [ ][ ][ ][ ][ ]
  • Ignore useradd warnings when building packages. [ ]
  • Update the script to powercycle armhf architecture nodes to add a hint to where nodes named virt-*. [ ]
  • Update the node health check to also fix failed logrotate and man-db services. [ ]
• Mattia Rizzolo:
  • Update the website job after contributors.sh script was rewritten in Python. [ ]
  • Make sure to set the DIFFOSCOPE environment variable when available. [ ]
• Vagrant Cascadian:
  • Various updates to the diffoscope timeouts. [ ][ ][ ]

Node maintenance was also performed by Holger Levsen [ ] and Vagrant Cascadian [ ].

Finally If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Twitter: @ReproBuilds
• Mailing list: rb-general@lists.reproducible-builds.org

I am going to start a new Linux focused FOSS online meeting for people in Australia and nearby areas. People can join from anywhere but the aim will be to support people in nearby areas. To cover the time zone range for Australia this requires a meeting on a weekend, I m thinking of the first Saturday of the month at 1PM Melbourne/Sydney time, that would be 10AM in WA and 3PM in NZ. We may have corner cases of daylight savings starting and ending on different days, but that shouldn t be a big deal as I think those times can vary by an hour either way without being too inconvenient for anyone. Note that I describe the meeting as Linux focused because my plans include having a meeting dedicated to different versions of BSD Unix and a meeting dedicated to the HURD. But those meetings will be mainly for Linux people to learn about other Unix-like OSs. One focus I want to have for the meetings is hands-on work, live demonstrations, and short highly time relevant talks. There are more lectures on YouTube than anyone could watch in a lifetime (see the Linux.conf.au channel for some good ones [1]). So I want to run events that give benefits that people can t gain from watching YouTube on their own. Russell Stuart and I have been kicking around ideas for this for a while. I think that the solution is to just do it. I know that Saturday won t work for everyone (no day will) but it will work for many people. I am happy to discuss changing the start time by an hour or two if that seems likely to get more people. But I m not particularly interested in trying to make it convenient for people in Hawaii or India, my idea is for an Australia/NZ focused event. I would be more than happy to share lecture notes etc with people in other countries who run similar events. As an aside I d be happy to give a talk for an online meeting at a Hawaiian LUG as the timezone is good for me. Please pencil in 1PM Melbourne time on the 5th of Feb for the first meeting. The meeting requirements will be a PC with good Internet access running a recent web browser and a ssh client for the hands-on stuff. A microphone or webcam is NOT required, any questions you wish to ask can be done with text if that s what you prefer. Suggestions for the name of the group are welcome.

• [1] https://www.youtube.com/c/linuxconfau/search

A friend recently reminded me of the existence of chrony, a "versatile implementation of the Network Time Protocol (NTP)". The excellent introduction is worth quoting in full:

It can synchronise the system clock with NTP servers, reference clocks (e.g. GPS receiver), and manual input using wristwatch and keyboard. It can also operate as an NTPv4 (RFC 5905) server and peer to provide a time service to other computers in the network. It is designed to perform well in a wide range of conditions, including intermittent network connections, heavily congested networks, changing temperatures (ordinary computer clocks are sensitive to temperature), and systems that do not run continuosly, or run on a virtual machine. Typical accuracy between two machines synchronised over the Internet is within a few milliseconds; on a LAN, accuracy is typically in tens of microseconds. With hardware timestamping, or a hardware reference clock, sub-microsecond accuracy may be possible.

Now that's already great documentation right there. What it is, why it's good, and what to expect from it. I want more. They have a very handy comparison table between chrony, ntp and openntpd.

My problem with OpenNTPd Following concerns surrounding the security (and complexity) of the venerable ntp program, I have, a long time ago, switched to using openntpd on all my computers. I hadn't thought about it until I recently noticed a lot of noise on one of my servers:

jan 18 10:09:49 curie ntpd[1069]: adjusting local clock by -1.604366s
jan 18 10:08:18 curie ntpd[1069]: adjusting local clock by -1.577608s
jan 18 10:05:02 curie ntpd[1069]: adjusting local clock by -1.574683s
jan 18 10:04:00 curie ntpd[1069]: adjusting local clock by -1.573240s
jan 18 10:02:26 curie ntpd[1069]: adjusting local clock by -1.569592s

You read that right, openntpd was constantly rewinding the clock, sometimes in less than two minutes. The above log was taken while doing diagnostics, looking at the last 30 minutes of logs. So, on average, one 1.5 seconds rewind per 6 minutes! That might be due to a dying real time clock (RTC) or some other hardware problem. I know for a fact that the CMOS battery on that computer (curie) died and I wasn't able to replace it (!). So that's partly garbage-in, garbage-out here. But still, I was curious to see how chrony would behave... (Spoiler: much better.) But I also had trouble on another workstation, that one a much more recent machine (angela). First, it seems OpenNTPd would just fail at boot time:

anarcat@angela:~(main)$ sudo systemctl status openntpd
  openntpd.service - OpenNTPd Network Time Protocol
     Loaded: loaded (/lib/systemd/system/openntpd.service; enabled; vendor pres>
     Active: inactive (dead) since Sun 2022-01-23 09:54:03 EST; 6h ago
       Docs: man:openntpd(8)
    Process: 3291 ExecStartPre=/usr/sbin/ntpd -n $DAEMON_OPTS (code=exited, sta>
    Process: 3294 ExecStart=/usr/sbin/ntpd $DAEMON_OPTS (code=exited, status=0/>
   Main PID: 3298 (code=exited, status=0/SUCCESS)
        CPU: 34ms
jan 23 09:54:03 angela systemd[1]: Starting OpenNTPd Network Time Protocol...
jan 23 09:54:03 angela ntpd[3291]: configuration OK
jan 23 09:54:03 angela ntpd[3297]: ntp engine ready
jan 23 09:54:03 angela ntpd[3297]: ntp: recvfrom: Permission denied
jan 23 09:54:03 angela ntpd[3294]: Terminating
jan 23 09:54:03 angela systemd[1]: Started OpenNTPd Network Time Protocol.
jan 23 09:54:03 angela systemd[1]: openntpd.service: Succeeded.

After a restart, somehow it worked, but it took a long time to sync the clock. At first, it would just not consider any peer at all:

anarcat@angela:~(main)$ sudo ntpctl -s all
0/20 peers valid, clock unsynced
peer
   wt tl st  next  poll          offset       delay      jitter
159.203.8.72 from pool 0.debian.pool.ntp.org
    1  5  2    6s    6s             ---- peer not valid ----
138.197.135.239 from pool 0.debian.pool.ntp.org
    1  5  2    6s    7s             ---- peer not valid ----
216.197.156.83 from pool 0.debian.pool.ntp.org
    1  4  1    2s    9s             ---- peer not valid ----
142.114.187.107 from pool 0.debian.pool.ntp.org
    1  5  2    5s    6s             ---- peer not valid ----
216.6.2.70 from pool 1.debian.pool.ntp.org
    1  4  2    2s    8s             ---- peer not valid ----
207.34.49.172 from pool 1.debian.pool.ntp.org
    1  4  2    0s    5s             ---- peer not valid ----
198.27.76.102 from pool 1.debian.pool.ntp.org
    1  5  2    5s    5s             ---- peer not valid ----
158.69.254.196 from pool 1.debian.pool.ntp.org
    1  4  3    1s    6s             ---- peer not valid ----
149.56.121.16 from pool 2.debian.pool.ntp.org
    1  4  2    5s    9s             ---- peer not valid ----
162.159.200.123 from pool 2.debian.pool.ntp.org
    1  4  3    1s    6s             ---- peer not valid ----
206.108.0.131 from pool 2.debian.pool.ntp.org
    1  4  1    6s    9s             ---- peer not valid ----
205.206.70.40 from pool 2.debian.pool.ntp.org
    1  5  2    8s    9s             ---- peer not valid ----
2001:678:8::123 from pool 2.debian.pool.ntp.org
    1  4  2    5s    9s             ---- peer not valid ----
2606:4700:f1::1 from pool 2.debian.pool.ntp.org
    1  4  3    2s    6s             ---- peer not valid ----
2607:5300:205:200::1991 from pool 2.debian.pool.ntp.org
    1  4  2    5s    9s             ---- peer not valid ----
2607:5300:201:3100::345c from pool 2.debian.pool.ntp.org
    1  4  4    1s    6s             ---- peer not valid ----
209.115.181.110 from pool 3.debian.pool.ntp.org
    1  5  2    5s    6s             ---- peer not valid ----
205.206.70.42 from pool 3.debian.pool.ntp.org
    1  4  2    0s    6s             ---- peer not valid ----
68.69.221.61 from pool 3.debian.pool.ntp.org
    1  4  1    2s    9s             ---- peer not valid ----
162.159.200.1 from pool 3.debian.pool.ntp.org
    1  4  3    4s    7s             ---- peer not valid ----

Then it would accept them, but still wouldn't sync the clock:

anarcat@angela:~(main)$ sudo ntpctl -s all
20/20 peers valid, clock unsynced
peer
   wt tl st  next  poll          offset       delay      jitter
159.203.8.72 from pool 0.debian.pool.ntp.org
    1  8  2    5s    6s         0.672ms    13.507ms     0.442ms
138.197.135.239 from pool 0.debian.pool.ntp.org
    1  7  2    4s    8s         1.260ms    13.388ms     0.494ms
216.197.156.83 from pool 0.debian.pool.ntp.org
    1  7  1    3s    5s        -0.390ms    47.641ms     1.537ms
142.114.187.107 from pool 0.debian.pool.ntp.org
    1  7  2    1s    6s        -0.573ms    15.012ms     1.845ms
216.6.2.70 from pool 1.debian.pool.ntp.org
    1  7  2    3s    8s        -0.178ms    21.691ms     1.807ms
207.34.49.172 from pool 1.debian.pool.ntp.org
    1  7  2    4s    8s        -5.742ms    70.040ms     1.656ms
198.27.76.102 from pool 1.debian.pool.ntp.org
    1  7  2    0s    7s         0.170ms    21.035ms     1.914ms
158.69.254.196 from pool 1.debian.pool.ntp.org
    1  7  3    5s    8s        -2.626ms    20.862ms     2.032ms
149.56.121.16 from pool 2.debian.pool.ntp.org
    1  7  2    6s    8s         0.123ms    20.758ms     2.248ms
162.159.200.123 from pool 2.debian.pool.ntp.org
    1  8  3    4s    5s         2.043ms    14.138ms     1.675ms
206.108.0.131 from pool 2.debian.pool.ntp.org
    1  6  1    0s    7s        -0.027ms    14.189ms     2.206ms
205.206.70.40 from pool 2.debian.pool.ntp.org
    1  7  2    1s    5s        -1.777ms    53.459ms     1.865ms
2001:678:8::123 from pool 2.debian.pool.ntp.org
    1  6  2    1s    8s         0.195ms    14.572ms     2.624ms
2606:4700:f1::1 from pool 2.debian.pool.ntp.org
    1  7  3    6s    9s         2.068ms    14.102ms     1.767ms
2607:5300:205:200::1991 from pool 2.debian.pool.ntp.org
    1  6  2    4s    9s         0.254ms    21.471ms     2.120ms
2607:5300:201:3100::345c from pool 2.debian.pool.ntp.org
    1  7  4    5s    9s        -1.706ms    21.030ms     1.849ms
209.115.181.110 from pool 3.debian.pool.ntp.org
    1  7  2    0s    7s         8.907ms    75.070ms     2.095ms
205.206.70.42 from pool 3.debian.pool.ntp.org
    1  7  2    6s    9s        -1.729ms    53.823ms     2.193ms
68.69.221.61 from pool 3.debian.pool.ntp.org
    1  7  1    1s    7s        -1.265ms    46.355ms     4.171ms
162.159.200.1 from pool 3.debian.pool.ntp.org
    1  7  3    4s    8s         1.732ms    35.792ms     2.228ms

It took a solid five minutes to sync the clock, even though the peers were considered valid within a few seconds:

jan 23 15:58:41 angela systemd[1]: Started OpenNTPd Network Time Protocol.
jan 23 15:58:58 angela ntpd[84086]: peer 142.114.187.107 now valid
jan 23 15:58:58 angela ntpd[84086]: peer 198.27.76.102 now valid
jan 23 15:58:58 angela ntpd[84086]: peer 207.34.49.172 now valid
jan 23 15:58:58 angela ntpd[84086]: peer 209.115.181.110 now valid
jan 23 15:58:59 angela ntpd[84086]: peer 159.203.8.72 now valid
jan 23 15:58:59 angela ntpd[84086]: peer 138.197.135.239 now valid
jan 23 15:58:59 angela ntpd[84086]: peer 162.159.200.123 now valid
jan 23 15:58:59 angela ntpd[84086]: peer 2607:5300:201:3100::345c now valid
jan 23 15:59:00 angela ntpd[84086]: peer 2606:4700:f1::1 now valid
jan 23 15:59:00 angela ntpd[84086]: peer 158.69.254.196 now valid
jan 23 15:59:01 angela ntpd[84086]: peer 216.6.2.70 now valid
jan 23 15:59:01 angela ntpd[84086]: peer 68.69.221.61 now valid
jan 23 15:59:01 angela ntpd[84086]: peer 205.206.70.40 now valid
jan 23 15:59:01 angela ntpd[84086]: peer 205.206.70.42 now valid
jan 23 15:59:02 angela ntpd[84086]: peer 162.159.200.1 now valid
jan 23 15:59:04 angela ntpd[84086]: peer 216.197.156.83 now valid
jan 23 15:59:05 angela ntpd[84086]: peer 206.108.0.131 now valid
jan 23 15:59:05 angela ntpd[84086]: peer 2001:678:8::123 now valid
jan 23 15:59:05 angela ntpd[84086]: peer 149.56.121.16 now valid
jan 23 15:59:07 angela ntpd[84086]: peer 2607:5300:205:200::1991 now valid
jan 23 16:03:47 angela ntpd[84086]: clock is now synced

That seems kind of odd. It was also frustrating to have very little information from ntpctl about the state of the daemon. I understand it's designed to be minimal, but it could inform me on his known offset, for example. It does tell me about the offset with the different peers, but not as clearly as one would expect. It's also unclear how it disciplines the RTC at all.

Compared to chrony Now compare with chrony:

jan 23 16:07:16 angela systemd[1]: Starting chrony, an NTP client/server...
jan 23 16:07:16 angela chronyd[87765]: chronyd version 4.0 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEBUG)
jan 23 16:07:16 angela chronyd[87765]: Initial frequency 3.814 ppm
jan 23 16:07:16 angela chronyd[87765]: Using right/UTC timezone to obtain leap second data
jan 23 16:07:16 angela chronyd[87765]: Loaded seccomp filter
jan 23 16:07:16 angela systemd[1]: Started chrony, an NTP client/server.
jan 23 16:07:21 angela chronyd[87765]: Selected source 206.108.0.131 (2.debian.pool.ntp.org)
jan 23 16:07:21 angela chronyd[87765]: System clock TAI offset set to 37 seconds

First, you'll notice there's none of that "clock synced" nonsense, it picks a source, and then... it's just done. Because the clock on this computer is not drifting that much, and openntpd had (presumably) just sync'd it anyways. And indeed, if we look at detailed stats from the powerful chronyc client:

anarcat@angela:~(main)$ sudo chronyc tracking
Reference ID    : CE6C0083 (ntp1.torix.ca)
Stratum         : 2
Ref time (UTC)  : Sun Jan 23 21:07:21 2022
System time     : 0.000000311 seconds slow of NTP time
Last offset     : +0.000807989 seconds
RMS offset      : 0.000807989 seconds
Frequency       : 3.814 ppm fast
Residual freq   : -24.434 ppm
Skew            : 1000000.000 ppm
Root delay      : 0.013200894 seconds
Root dispersion : 65.357254028 seconds
Update interval : 1.4 seconds
Leap status     : Normal

We see that we are nanoseconds away from NTP time. That was ran very quickly after starting the server (literally in the same second as chrony picked a source), so stats are a bit weird (e.g. the Skew is huge). After a minute or two, it looks more reasonable:

Reference ID    : CE6C0083 (ntp1.torix.ca)
Stratum         : 2
Ref time (UTC)  : Sun Jan 23 21:09:32 2022
System time     : 0.000487002 seconds slow of NTP time
Last offset     : -0.000332960 seconds
RMS offset      : 0.000751204 seconds
Frequency       : 3.536 ppm fast
Residual freq   : +0.016 ppm
Skew            : 3.707 ppm
Root delay      : 0.013363549 seconds
Root dispersion : 0.000324015 seconds
Update interval : 65.0 seconds
Leap status     : Normal

Now it's learning how good or bad the RTC clock is ("Frequency"), and is smoothly adjusting the System time to follow the average offset (RMS offset, more or less). You'll also notice the Update interval has risen, and will keep expanding as chrony learns more about the internal clock, so it doesn't need to constantly poll the NTP servers to sync the clock. In the above, we're 487 micro seconds (less than a milisecond!) away from NTP time. (People interested in the explanation of every single one of those fields can read the excellent chronyc manpage. That thing made me want to nerd out on NTP again!) On the machine with the bad clock, chrony also did a 1.5 second adjustment, but just once, at startup:

jan 18 11:54:33 curie chronyd[2148399]: Selected source 206.108.0.133 (2.debian.pool.ntp.org) 
jan 18 11:54:33 curie chronyd[2148399]: System clock wrong by -1.606546 seconds 
jan 18 11:54:31 curie chronyd[2148399]: System clock was stepped by -1.606546 seconds 
jan 18 11:54:31 curie chronyd[2148399]: System clock TAI offset set to 37 seconds 

Then it would still struggle to keep the clock in sync, but not as badly as openntpd. Here's the offset a few minutes after that above startup:

System time     : 0.000375352 seconds slow of NTP time

And again a few seconds later:

System time     : 0.001793046 seconds slow of NTP time

I don't currently have access to that machine, and will update this post with the latest status, but so far I've had a very good experience with chrony on that machine, which is a testament to its resilience, and it also just works on my other machines as well.

Extras On top of "just working" (as demonstrated above), I feel that chrony's feature set is so much superior... Here's an excerpt of the extras in chrony, taken from the comparison table:

• source frequency tracking
• source state restore from file
• temperature compensation
• ready for next NTP era (year 2036)
• replace unreachable / falseticker servers
• aware of jitter
• RTC drift tracking
• RTC trimming
• Restore time from file w/o RTC
• leap seconds correction, in slew mode
• drops root privileges

I even understand some of that stuff. I think. So kudos to the chrony folks, I'm switching.

Caveats One thing to keep in mind in the above, however is that it's quite possible chrony does as bad of a job as openntpd on that old machine, and just doesn't tell me about it. For example, here's another log sample from another server (marcos):

jan 23 11:13:25 marcos ntpd[1976694]: adjusting clock frequency by 0.451035 to -16.420273ppm

I get those basically every day, which seems to show that it's at least trying to keep track of the hardware clock. In other words, it's quite possible I have no idea what I'm talking about and you definitely need to take this article with a grain of salt. I'm not an NTP expert. Update: I should also mentioned that I haven't evaluated systemd-timesyncd, for a few reasons:

1. I have enough things running under systemd
2. I wasn't aware of it when I started writing this
3. I couldn't find good documentation on it... later I found the above manpage and of course the Arch Wiki but that is very minimal
4. therefore I can't tell how it compares with chrony or (open)ntpd, so I don't see an enticing reason to switch

It has a few things going for it though:

• it's likely shipped with your distribution already
• it drops privileges (possibly like chrony, unclear if it also has seccomp filters)
• it's minimalist: it only does SNTP so not the server side
• the status command is good enough that you can tell the clock frequency, precision, and so on (especially when compared to openntpd's ntpctl)

So I'm reserving judgement over it, but I'd certainly note that I'm always a little weary in trusting systemd daemons with the network, and would prefer to keep that attack surface to a minimum. Diversity is a good thing, in general, so I'll keep chrony for now. It would certainly nice to see it added to chrony's comparison table.

Switching to chrony Because the default configuration in chrony (at least as shipped in Debian) is sane (good default peers, no open network by default), installing it is as simple as:

apt install chrony

And because it somehow conflicts with openntpd, that also takes care of removing that cruft as well.

Update: Debian defaults So it seems like I managed to write this entire blog post without putting it in relation with the original reason I had to think about this in the first place, which is odd and should be corrected. This conversation came about on an IRC channel that mentioned that the ntp package (and upstream) is in bad shape in Debian. In that discussion, chrony and ntpsec were discussed as possible replacements, but when we had the discussion on chat, I mentioned I was using openntpd, and promptly realized I was actually unhappy with it. A friend suggested chrony, I tried it, and it worked amazingly, I switched, wrote this blog post, end of story. Except today (2022-02-07, two weeks later), I actually read that thread and realized that something happened in Debian I wasn't actually aware of. In bookworm, systemd-timesyncd was not only shipped, but it was installed by default, as it was marked as a hard dependency of systemd. That was "fixed" in systemd-247.9-2 (see bug 986651), but only by making the dependency a Recommends and marking it as Priority: important. So in effect, systemd-timesyncd became the default NTP daemon in Debian in bookworm, which I find somewhat surprising. timesyncd has many things going for it (as mentioned above), but I do find it a bit annoying that systemd is replacing all those utilities in such a way. I also wonder what is going to happen on upgrades. This is all a little frustrating too because there is no good comparison between the other NTP daemons and timesyncd anywhere. The chrony comparison table doesn't mention it, and an audit by the Core Infrastructure Initiative from 2017 doesn't mention it either, even though timesyncd was announced in 2014. (Same with this blog post from Facebook.)

Several issues were brought before the Debian Community team regarding responsiveness, tone, and needed software updates to forums.debian.net. The question was asked, who s in charge? Over the course of the discussion several Debian Developers volunteered to help by providing a presence on the forums from Debian and to assist with the necessary changes to keep the service up and running. We are happy to announce the following changes to the (NEW!) forums.debian.net, which have and should address most of the prior concerns with accountability, tone, use, and reliability: Debian Developers: Paulo Henrique de Lima Santana (phls), Felix Lechner (lechner), and Donald Norwood (donald) have been added to the forum's Server and Administration teams. The server instance is now running directly within Debian's infrastructure. The forum software and back-end have been updated to the most recent versions where applicable. DNS resolves for both IPv4 and IPv6. SSL/HTTPS are enabled. (It s 2021!) New Captcha and Anti-spam systems are in place to thwart spammers, bots, and to make it easier for humans to register. New Administrators and Moderation staff were added to provide additional coverage across the hours and to combine years of experience with forum operation and Debian usage. New viewing styles are available for users to choose from, some of which are ideal for mobile/tablet viewing. We inadvertently fixed the time issue that the prior forum had of running 11 minutes fast. :) We have clarified staff roles and staff visibility. Responsiveness to users on the forums has increased. Email addresses for mods/admins have been updated and checked for validity, it has seen direct use and response. The guidelines for forum use by users and staff have been updated. The Debian COC has been made into a Global Announcement as an accompanyist to the newly updated guidelines to give the moderators/administrators an additional rule-set for unruly or unbecoming behavior. Some of the discussion areas have been renamed and refocused, along with the movement of multiple threads to make indexing and searching of the forums easier. Many (New!) features and extensions have been added to the forum for ease of use and modernization, such as a user thanks system and thread hover previews. There are some server administrative tasks that were upgraded as well which don't belong on a public list, but we are backing up regularly and secure. :) We have a few minor details here and there to attend to and the work is ongoing. Many Thanks and Appreciation to the Debian System Administrators (DSA) and Ganneff who took the time to coordinate and assist with the instance, DNS, and network and server administration minutiae, our helpful DPL Jonathan Carter, many thanks to the current and prior forum moderators and administrators: Mez, sunrat, 4D696B65, arochester, and cds60601 for helping with the modifications and transition, and to the forum users who participated in lots of the tweaking. All in all this was a large community task and everyone did a significant part. Thank you!

Tonight I m provisioning a new virtual machine at Hetzner and I wanted to share how Consfigurator is helping with that. Hetzner have a Debian buster image you can start with, as you d expect, but it comes with things like cloud-init, preconfiguration to use Hetzner s apt mirror which doesn t serve source packages(!), and perhaps other things I haven t discovered. It s a fine place to begin, but I want all the configuration for this server to be explicit in my Consfigurator consfig, so it is good to start with pristine upstream Debian. I could boot one of Hetzner s installation ISOs but that s slow and manual. Consfigurator can replace the OS in the VM s root filesystem and reboot for me, and we re ready to go. Here s the configuration:

(defhost foo.silentflame.com (:deploy ((:ssh :user "root") :sbcl))
  (os:debian-stable "buster" :amd64)
  ;; Hetzner's Debian 10 image comes with a three-partition layout and boots
  ;; with traditional BIOS.
  (disk:has-volumes
   (physical-disk
    :device-file "/dev/sda" :boots-with '(grub:grub :target "i386-pc")))
  (on-change (installer:cleanly-installed-once
              nil
              ;; This is a specification of the OS Hetzner's image has, so
              ;; Consfigurator knows how to install SBCL and debootstrap(8).
              ;; In this case it's the same Debian release as the replacement.
              '(os:debian-stable "buster" :amd64))
    ;; Clear out the old OS's EFI system partition contents, in case we can
    ;; switch to booting with EFI at some point (if we wanted we could specify
    ;; an additional x86_64-efi target above, and grub-install would get run
    ;; to repopulate /boot/efi, but I don't think Hetzner can boot from it yet).
    (file:directory-does-not-exist "/boot/efi/EFI")
    (apt:installed "linux-image-amd64")
    (installer:bootloaders-installed)
    (fstab:entries-for-volumes
     (disk:volumes
       (mounted-ext4-filesystem :mount-point "/")
       (partition
        (mounted-fat32-filesystem
         :mount-options '("umask=0077") :mount-point "/boot/efi"))))
    (file:lacks-lines "/etc/fstab" "# UNCONFIGURED FSTAB FOR BASE SYSTEM")
    (file:is-copy-of "/etc/resolv.conf" "/old-os/etc/resolv.conf")
    (mount:unmounted-below-and-removed "/old-os"))
  (apt:mirror "http://ftp.de.debian.org/debian")
  (apt:no-pdiffs)
  (apt:standard-sources.list)
  (sshd:installed)
  (as "root" (ssh:authorized-keys +spwsshkey+))
  (sshd:no-passwords)
  (timezone:configured "Etc/UTC")
  (swap:has-swap-file "2G")
  (network:clean-/etc/network/interfaces)
  (network:static "enp1s0" "xxx.xxx.xxx.xxx" "xxx.xxx.1.1" "255.255.255.255"))

and to use it you evaluate this at the REPL:

CONSFIG> (deploy ((:ssh :user "root" :hop "xxx.xxx.xxx.xxx") :sbcl) foo.silentflame.com)

Here the :HOP parameter specifies the IP address of the new machine, as DNS hasn t been updated yet. Consfigurator installs SBCL and debootstrap(8), prepares a minimal system, replaces the contents of /, gets to work applying the other properties, and then reboots. This gets us a properly populated fstab:

UUID=...            /           ext4    relatime    0   1
PARTUUID=...        /boot/efi   vfat    umask=0077  0   2
/var/lib/swapfile   swap        swap    defaults    0   0

(slightly doctored for more readable alignment) There s ordering logic so that the swapfile will end up after whatever filesystem contains it; a UUID is used for ext4 filesystems, but for fat32 filesystems, to be safe, a PARTUUID is used. The application of (INSTALLER:BOOTLOADERS-INSTALLED) handles calling both update-grub(8) and grub-install(8), relying on the metadata specified about /dev/sda. Next time we execute Consfigurator against the machine, it ll ignore all the property applications attached to the application of (INSTALLER:CLEANLY-INSTALLED-ONCE) with ON-CHANGE, and just apply everything following that block. There are a few things I don t have good solutions for. When you boot Hetzner s image the primary network interface is eth0, but then for a freshly debootstrapped Debian you get enp1s0, and I haven t got a good way of knowing what it ll be (if you know it ll have the same name, you can use (NETWORK:PRESERVE-STATIC-ONCE) to create a file in /etc/network/interfaces.d based on the current default route and corresponding interface). Another tricky thing is SSH host keys. It s easy to use Consfigurator to add host keys to your laptop s ~/.ssh/known_hosts, but in this case the host key changes back and forth from whatever the Hetzner image has and the newly generated key you get afterwards. One option might be to copy the old host keys out of /old-os before it gets deleted, like how /etc/resolv.conf is copied. This work is based on Propellor s equivalent functionality. I think my approach to handling /etc/fstab and bootloader installation is an improvement on what Joey does.

It's been a long while since I last hosted a BSP, but 'tis the season. Kubernetes SIG Node will be holding a bug scrub on June 24-25, and this is a great opportunity for you to get involved if you're interested in contributing to Kubernetes or SIG Node! We will be hosting a global event with region captains for all timezones. I am one of the NASA captains (~17:00-01:00 UTC) and I'll be leading the kickoff. We will be working on Slack and Zoom. I hope you'll be able to drop in! Details

• Date: Thursday, June 24 through Friday, June 25
• Start: 00:00 UTC June 24
• End: 23:59 UTC June 25
• Sign up: Participants do not have to sign up, just show up!
• Slack room: #sig-node-bug-scrub on Kubernetes Slack
• Docs: See the HackMD docs.
• Email: See the kubernetes-dev mailing list.

I'm an existing contributor, what should I work on? Work on triaging and closing SIG Node bugs. We have a lot of bugs!! The goal of our event is to categorize, clean up, and resolve some of the 450+ issues in k/k for SIG Node. Check out the event docs for more instructions. I'm a new contributor and want to join but I have no idea what I'm doing! At some point, that was all of us! This is a great opportunity to get involved if you've never contributed to Kubernetes. We'll have dedicated mentors available to coordinate and help out new contributors. If you've never contributed to Kubernetes before, I recommend you check out the Getting Started and Contributor Guide resources in advance of the event. You will want to ensure you've signed the contributor license agreement (CLA). Remember, you don't have to code to make valuable contributions! Triaging the bug tracker is a great example of this. See you there! Happy hacking.

The DebConf team is glad to announce that registration for DebConf21 Online is now open. The 21st Debian Conference is being held Online, due to COVID-19, from August 22 to August 29, 2021. It will also sport a DebCamp from August 15 to August 21, 2021 (preceeding the DebConf). To register for DebConf21, please visit the DebConf website at https://debconf21.debconf.org/register Reminder: Creating an account on the site does not register you for the conference, there's a conference registration form to complete after signing in. Participation in DebConf21 is conditional on your respect of our Code of Conduct. We require you to read, understand and abide by this code. A few notes about the registration process:

• We need to know attendees' locations to better plan the schedule around timezones. Please make sure you fill in the "Country I call home" field in the registration form accordingly. It's especially important to have this data for people who submitted talks, but also for other attendees.
• We are offering limited amounts of financial support for those who require it in order to attend. Please refer to the corresponding page on the website for more information.

Any questions about registration should be addressed to registration@debconf.org. See you online! DebConf would not be possible without the generous support of all our sponsors, especially our Platinum Sponsors Lenovo and Infomaniak, and our Gold Sponsor Matanel Foundation.

Powerlevel10k is a theme for Zsh. It contains some powerful features, is astoundingly fast, and easy to customize. I am quite amazed at the skills of its main author. Be sure to also have a look at Zsh for Humans, a complete Zsh configuration including this theme. One of the nice features of Powerlevel10k is transient prompts: past prompts are reduced to a more minimal configuration to save space by removing unneeded information.

My implementation of a transient prompt with Zsh. Past prompts are compact and include the time of the command execution, the hostname, and the status of the previous command while the complete prompt contains more information like the current directory and the Git branch.

When it comes to configuring my shell, I still prefer writing and understanding each line going into it. Therefore, I am still building my Zsh configuration from scratch. Here is how I have integrated the above transient feature into my prompt. The first step is to configure the appearance of the prompt in its compact form. Let s assume we have a variable, $_vbe_prompt_compact set to 1 when we want a compact prompt. We use the following function to define the prompt appearance: Our next step is to redraw the prompt after accepting a command. We wrap Zsh line editor into a function:¹ That s all!

---

One downside of using the powerline fonts is that it messes with copy/paste. As I am using tmux, I use the following snippet to work around this issue and use only standard Unicode characters when copying from the terminal: Copying and pasting the text from the screenshot above yields the following text:

Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 865 other packages on CRAN. This new release brings Armadillo 10.5.0 which was released early on Friday. We had done one full test in the 10.5 rc1 prerelease one week earlier, and did another test on 10.5.0 and this 0.10.5.0.0 RcppArmadillo release just for added rigour. The package was then uploaded to CRAN late Friday (my timezone). The automated process flagged one NOTE as a false positive (yet another instance of the well-known (yet dreaded) issue of Suggests != Depends by one these 865 packages). This lead to a need of an inspection by one of the CRAN maintainers, and the weekend being the weekend it was only processed just now. Upstream moves at a speed that is a little faster than the cadence CRAN likes. As we had released RcppArmadillo 0.10.4.0.0 on April 13 we did not want to follow-up too soon thereafter with 0.10.4.1.0 which was thusly only a GitHub and drat release (which can always be had easily too via install.packages("RcppArmadillo", repos="https://RcppCore.github.io/drat").) The full set of changes follows. We include the aforementioned interim release as well.

Changes in RcppArmadillo version 0.10.5.0 (2021-05-21)

• Upgraded to Armadillo release 10.5 (Antipodean Fortress)
  • added .clamp() member function
  • expanded the standalone clamp() function to handle complex values
  • more efficient use of OpenMP
  • vector, matrix and cube constructors now initialise elements to zero by default; use the fill::none specifier, eg. mat X(4,5,fill::none), to disable element initialisation
• Added codecov.yml to exclude Armadillo from coverage analysis

Changes in RcppArmadillo version 0.10.4.1.0 (2021-04-23)

• Upgraded to Armadillo release 10.4.1 (Pressure Cooker)
• GitHub-only release

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

I wrote for LWN for about two years. During that time, I wrote (what seems to me an impressive) 34 articles, but I always had a pile of ideas in the back of my mind. Those are ideas, notes, and scribbles lying around. Some were just completely abandoned because they didn't seem a good fit for LWN. Concretely, I stored those in branches in a git repository, and used the branch name (and, naively, the last commit log) as indicators of the topic. This was the state of affairs when I left:

remotes/private/attic/novena                    822ca2bb add letter i sent to novena, never published
remotes/private/attic/secureboot                de09d82b quick review, add note and graph
remotes/private/attic/wireguard                 5c5340d1 wireguard review, tutorial and comparison with alternatives
remotes/private/backlog/dat                     914c5edf Merge branch 'master' into backlog/dat
remotes/private/backlog/packet                  9b2c6d1a ham radio packet innovations and primer
remotes/private/backlog/performance-tweaks      dcf02676 config notes for http2
remotes/private/backlog/serverless              9fce6484 postponed until kubecon europe
remotes/private/fin/cost-of-hosting             00d8e499 cost-of-hosting article online
remotes/private/fin/kubecon                     f4fd7df2 remove published or spun off articles
remotes/private/fin/kubecon-overview            21fae984 publish kubecon overview article
remotes/private/fin/kubecon2018                 1edc5ec8 add series
remotes/private/fin/netconf                     3f4b7ece publish the netconf articles
remotes/private/fin/netdev                      6ee66559 publish articles from netdev 2.2
remotes/private/fin/pgp-offline                 f841deed pgp offline branch ready for publication
remotes/private/fin/primes                      c7e5b912 publish the ROCA paper
remotes/private/fin/runtimes                    4bee1d70 prepare publication of runtimes articles
remotes/private/fin/token-benchmarks            5a363992 regenerate timestamp automatically
remotes/private/ideas/astropy                   95d53152 astropy or python in astronomy
remotes/private/ideas/avaneya                   20a6d149 crowdfunded blade-runner-themed GPLv3 simcity-like simulator
remotes/private/ideas/backups-benchmarks        fe2f1f13 review of backup software through performance and features
remotes/private/ideas/cumin                     7bed3945 review of the cumin automation tool from WM foundation
remotes/private/ideas/future-of-distros         d086ca0d modern packaging problems and complex apps
remotes/private/ideas/on-dying                  a92ad23f another dying thing
remotes/private/ideas/openpgp-discovery         8f2782f0 openpgp discovery mechanisms (WKD, etc), thanks to jonas meurer
remotes/private/ideas/password-bench            451602c0 bruteforce estimates for various password patterns compared with RSA key sizes
remotes/private/ideas/prometheus-openmetrics    2568dbd6 openmetrics standardizing prom metrics enpoints
remotes/private/ideas/telling-time              f3c24a53 another way of telling time
remotes/private/ideas/wallabako                 4f44c5da talk about wallabako, read-it-later + kobo hacking
remotes/private/stalled/bench-bench-bench       8cef0504 benchmarking http benchmarking tools
remotes/private/stalled/debian-survey-democracy 909bdc98 free software surveys and debian democracy, volunteer vs paid work

Wow, what a mess! Let's see if I can make sense of this:

Attic Those are articles that I thought about, then finally rejected, either because it didn't seem worth it, or my editors rejected it, or I just moved on:

• novena: the project is ooold now, didn't seem to fit a LWN article. it was basically "how can i build my novena now" and "you guys rock!" it seems like the MNT Reform is the brain child of the Novena now, and I dare say it's even cooler!
• secureboot: my LWN editors were critical of my approach, and probably rightly so - it's a really complex subject and I was probably out of my depth... it's also out of date now, we did manage secureboot in Debian
• wireguard: LWN ended up writing extensive coverage, and I was biased against Donenfeld because of conflicts in a previous project

Backlog Those were articles I was planning to write about next.

• dat: I already had written Sharing and archiving data sets with Dat, but it seems I had more to say... mostly performance issues, beaker, no streaming, limited adoption... to be investigated, I guess?
• packet: a primer on data communications over ham radio, and the cool new tech that has emerged in the free software world. those are mainly notes about Pat, Direwolf, APRS and so on... just never got around to making sense of it or really using the tech...
• performance-tweaks: "optimizing websites at the age of http2", the unwritten story of the optimization of this website with HTTP/2 and friends
• serverless: god. one of the leftover topics at Kubecon, my notes on this were thin, and the actual subject, possibly even thinner... the only lie worse than the cloud is that there's no server at all! concretely, that's a pile of notes about Kubecon which I wanted to sort through. Probably belongs in the attic now.

Fin Those are finished articles, they were published on my website and LWN, but the branches were kept because previous drafts had private notes that should not be published.

Ideas

• astropy: "Python in astronomy" - had a chat with saimn while writing about sigal, and it turns out he actually works on free software in astronomy, in Python... I actually expect LWN to cover this sooner than later, after Lee Phillips's introduction to SciPy
• avaneya: crowdfunded blade-runner-themed GPLv3 simcity-like simulator, i just have that link so far
• backups-benchmarks: review of backup software through performance and features, possibly based on those benchmarks, maybe based on this list from restic although they refused casync. benchmark articles are hard though, especially when you want to "cover them all"... I did write a silly Attic vs Bup back when those programs existed (2014), in a related note...
• ideas/cumin: review of the Cumin automation tool from WikiMedia Foundation... I ended up using the tool at work and writing service documentation for it
• ideas/future-of-distros: modern packaging problems and complex apps, starting from this discussion about the removal of Dolibarr from Debian, a summary of the thread from liw, and ideas from joeyh (now from the outside of Debian), then debates over the power of FTP masters - ugh, glad I didn't step in that rat's nest
• ideas/on-dying: "what happens when a hacker dies?" rather grim subject, but a more and more important one... joeyh has ideas again, phk as well, then there's a protocol for dying (really grim)... then there are site policies like GitHub, Facebook, etc... more in the branch, but that one I can't help but think about now that family has taken a bigger place in my life...
• ideas/openpgp-discovery: OpenPGP discovery mechanisms (WKD, etc), suggested by Jonas Meurer (somewhere?), only links to Mailveloppe, LEAP, WKD (or is it WKS?), another standard, probably would need to talk about OpenPGP CA now and how Debian and Tor manage their keyrings... pain in the back.
• ideas/password-bench: bruteforce estimates for various password patterns compared with RSA key sizes, spinoff of my smartcard article, in the crypto-bench, look at this shiny graph, surely that must mean an article, right?
• ideas/prometheus-openmetrics: "Evolving the Prometheus exposition format into a standard", seems like this happened
• ideas/telling-time: telling time to users is hard. xclock vs ttyclock, etc. maybe gameclock and undertime as well? syncing time is hard, but it turns out showing it is non trivial as well... basically turning this bug report into an article. for some reason I linked to this meme, derived from this meme, presumably a premonition of my stupid idea of writing undertime TIMEZONES!
• ideas/wallabako: "talk about wallabako, read-it-later + kobo hacking", that's it, not even a link to the project!

A lot of those branches were actually just an empty commit, with the commitlog being the "pitch", more or less. I'd send that list to my editors, sometimes with a few more links (basically the above), and they would nudge me one way or the other. Sometimes they would actively discourage me to write about something, and I would do it anyways, send them a draft, and they would patiently make me rewrite it until it was a decent article. This was especially hard with the terminal emulator series, which took forever to write and even got my editors upset when they realized I had never installed Fedora (I ended up installing it, and I was proven wrong!)

Stalled Oh, and then there's those: those are either "ideas" or "backlog" that got so far behind that I just moved them out of the way because I was tired of seeing them in my list.

• stalled/bench-bench-bench benchmarking http benchmarking tools, a horrible mess of links, copy-paste from terminals, and ideas about benchmarking... some of this trickled out into this benchmarking guide at Tor, but not much more than the list of tools
• stalled/debian-survey-democracy: "free software surveys and Debian democracy, volunteer vs paid work"... A long standing concern of mine is that all Debian work is supposed to be volunteer, and paying explicitly for work inside Debian has traditionally been frowned upon, even leading to serious drama and dissent (remember Dunc-Tank)? back when I was writing for LWN, I was also doing paid work for Debian LTS. I also learned that a lot (most?) Debian Developers were actually being paid by their job to work on Debian. So I was confused by this apparent contradiction, especially given how the LTS project has been mostly accepted, while Dunc-Tank was not... See also this talk at Debconf 16. I had hopes that this study would show the "hunch" people have offered (that most DDs are paid to work on Debian) but it seems to show the reverse (only 36% of DDs, and 18% of all respondents paid). So I am still confused and worried about the sustainability of Debian.

What do you think? So that's all I got. As people might have noticed here, I have much less time to write these days, but if there's any subject in there I should pick, what is the one that you would find most interesting? Oh! and I should mention that you can write to LWN! If you think people should know more about some Linux thing, you can get paid to write for it! Pitch it to the editors, they won't bite. The worst that can happen is that they say "yes" and there goes two years of your life learning to write. Because no, you don't know how to write, no one does. You need an editor to write. That's why this article looks like crap and has a smiley.

Wrote a timezone checker page. timezone. Shows the current time in blue line. I haven't made anything configurable but will think about it later.

So we're a fair way through the release, then. Testing of almost all the standard images has finished. Pretty much all of the disk images are now complete and in place.People are working their way through the tests of the debian-live images in the various desktop flavours. These have to be done on real hardware - so it does take time. A new tester - peylight - has dropped in to help for the first time. Sqrt not has also joined us from the other end of the timezone scale - we have somebody at UTC-0700 and somebody at UTC+0430 today. [I can't remember where Linux-fan is timezone wise] All of the help from all the testers is very welcome, as ever.
A slight pause - a couple of us have a meal to eat - but it looks as if we've done well on timings. The original estimate was for 2000UTC - maybe a little after that and we'll be finished and the images release can be published - there were a couple of minor hiccups but we've done well so far.
Thanks, as always, to the people behind the scenes doing all the work, to DSA and admins providing large machines for us to do the builds on and to the people who drop in and spend a few hours of their time on a working day/weekend to help out.

Here's how I set up a media PC using Kodi (formerly XMBC) and a Raspberry Pi 4.

Hardware The hardware is fairly straightforward, but here's what I ended up getting:

• Raspberry Pi 4 board: I went for the maximum amount of RAM (8 GB).
• SD-card: Since I'm not going to store any media on here, 32 GB is plenty.
• HDMI to micro-HDMI cable
• Case and power supply

You'll probably want to add a remote control to that setup. I used an old Streamzap I had lying around.

Installing the OS on the SD-card Plug the SD card into a computer using a USB adapter. Download the imager and use it to install Raspbian on the SDcard. Then you can simply plug the SD card into the Pi and boot.

System configuration Using sudo raspi-config, I changed the following:

• Set hostname (System Options)
• Wait for network at boot (System Options): needed for NFS
• Disable screen blanking (Display Options)
• Enable ssh (Interface Options)
• Configure locale, timezone and keyboard (Localisation Options)
• Set WiFi country (Localisation Options)

Then I enabled automatic updates:

apt install unattended-upgrades anacron
echo 'Unattended-Upgrade::Origins-Pattern  
        "origin=Debian,codename=$ distro_codename ,label=Debian";
        "origin=Debian,codename=$ distro_codename ,label=Debian-Security";
        "origin=Raspbian,codename=$ distro_codename ,label=Raspbian";
        "origin=Raspberry Pi Foundation,codename=$ distro_codename ,label=Raspberry Pi Foundation";
 ;'   sudo tee /etc/apt/apt.conf.d/51unattended-upgrades-raspbian

Headless setup Should you need to do the setup without a monitor, you can enable ssh by inserting the SD card into a computer and then creating an empty file called ssh in the boot partition. Plug it into your router and boot it up. Check the IP that it received by looking at the active DHCP leases in your router's admin panel. Then login:

ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no pi@192.168.1.xxx

using the default password of raspberry.

Hardening In order to secure the Pi, I followed most of the steps I usually take when setting up a new Linux server. I created a new user account for admin and ssh access:

adduser francois
addgroup sshuser
adduser francois sshuser
adduser francois sudo

and changed the pi user password to a random one:

pwgen -sy 32
sudo passwd pi

before removing its admin permissions:

deluser pi adm
deluser pi sudo
deluser pi dialout
deluser pi cdrom
deluser pi lpadmin

Finally, I enabled the Uncomplicated Firewall by installing its package:

apt install ufw

and only allowing ssh connections. After starting ufw using systemctl start ufw.service, you can check that it's configured as expected using ufw status. It should display the following:

Status: active
To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere
22/tcp (v6)                ALLOW       Anywhere (v6)

Installing Kodi Kodi is very straightforward to install since it's now part of the Raspbian repositories:

apt install kodi

To make it start at boot/login, while still being able to exit and use other apps if needed:

cp /etc/xdg/lxsession/LXDE-pi/autostart ~/.config/lxsession/LXDE-pi/
echo "@kodi" >> ~/.config/lxsession/LXDE-pi/autostart

Network File System In order to avoid having to have all media storage connected directly to the Pi via USB, I setup an NFS share over my local network. First, give static IP allocations to the server and the Pi in your DHCP server, then add it to the /etc/hosts file on your NFS server:

192.168.1.3    pi

Install the NFS server package:

apt instal nfs-kernel-server

Setup the directories to share in /etc/exports:

/pub/movies    pi(ro,insecure,all_squash,subtree_check)
/pub/tv_shows  pi(ro,insecure,all_squash,subtree_check)

Open the right ports on your firewall by putting this in /etc/network/iptables.up.rules:

-A INPUT -s 192.168.1.3 -p udp -j ACCEPT
-A INPUT -s 192.168.1.0/24 -p tcp --dport 111 -j ACCEPT
-A INPUT -s 192.168.1.0/24 -p udp --dport 111 -j ACCEPT
-A INPUT -s 192.168.1.0/24 -p udp --dport 123 -j ACCEPT
-A INPUT -s 192.168.1.0/24 -p tcp --dport 600:1124 -j ACCEPT
-A INPUT -s 192.168.1.0/24 -p udp --dport 600:1124 -j ACCEPT
-A INPUT -s 192.168.1.0/24 -p tcp --dport 2049 -j ACCEPT
-A INPUT -s 192.168.1.0/24 -p udp --dport 2049 -j ACCEPT

Finally, apply all of these changes:

iptables-apply
systemctl restart nfs-kernel-server.service

On the Pi, put the server's static IP in /etc/hosts:

192.168.1.2    fileserver

and this in /etc/fstab:

fileserver:/data/movies  /kodi/movies  nfs  ro,bg,hard,noatime,async,nolock  0  0
fileserver:/data/tv      /kodi/tv      nfs  ro,bg,hard,noatime,async,nolock  0  0

Then create the mount points and mount everything:

mkdir -p /kodi/movies
mkdir /kodi/tv
mount /kodi/movies
mount /kodi/tv

Here s my (sixteenth) monthly update about the activities I ve done in the F/L/OSS world.

Debian

This was my 25th month of contributing to Debian. I became a DM in late March 2019 and a DD on Christmas 19! \o/ This month was bat-shit crazy. Why? We ll come to it later, probably 15th of this month?
Anyway, besides being crazy, hectic, adventerous, and the first of 2021, this month I was super-insanely busy. With what? Hm, more about this later this month! ^_^ However, I still did some Debian stuff here and there. Here are the following things I worked on:

Uploads and bug fixes:

• ruby-rack (2.1.1-6) - Fixing CVE-2020-8184/bug #963477.
• ruby-faye-websocket (0.11.0-1) - Fixing CVE-2020-15133/bug #967061.
• ruby-faye (1.4.0-1) - Fixing CVE-2020-11020/bug #959392 and CVE-2020-15134/bug #967063.
• ruby-rack (2.1.4-2) - Fix failing tests and new upstream version, v2.1.4.
• ruby-rake-ant (1.0.4-1) - Initial release, bug #979498.
• libgit2 (1.1.0+dfsg.1-4) - Source-only upload for migration.
• ruby-scanf (1.0.0-1) - Initial release, bug #979497.
• polybar (3.5.4-1) - New upstream version, v3.5.4.
• fpc (3.2.0+dfsg-10) - Severe crash fix for bugs #979850, #979853, #979862, and #979851.
• ruby-em-redis (0.3.0+gh-3) - Fixing FTBFS, bug #978975, as requested by Holger! :)
• gdebi (0.9.5.7+nmu4) - Fixing FTBFS, bug #951923, as requested by Holger! :)
• ruby-redcarpet (3.4.0-4+deb10u1) - Fixing CVE-2020-26298/bug #980057.
• ruby-in-parallel (0.1.17-1.2) - Fixing autopkgtest, bug #979700.
• ruby-in-parallel (0.1.17-1.3) - Fixing random test failures, bug #980585.
• python-bottle (0.12.15-2+deb10u1) - Fixing CVE-2020-28473.

Other $things:

• Attended the Debian Ruby team meeting.
• Mentoring for newcomers.
• Moderation of -project mailing list.
• Sponsored golang-github-gorilla-css for Fedrico.

---

Debian (E)LTS

Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success. And Debian Extended LTS (ELTS) is its sister project, extending support to the Jessie release (+2 years after LTS support). This was my sixteenth month as a Debian LTS and seventh month as a Debian ELTS paid contributor.
I was assigned 26.00 hours for LTS and 36.75 hours for ELTS and worked on the following things:
(however, I worked extra for 9 hours for LTS and 9 hours for ELTS this month, which I intend to balance from the next month!)

LTS CVE Fixes and Announcements:

• Issued DLA 2518-1, fixing CVE-2020-35492, for cairo.
  For Debian 9 Stretch, these problems have been fixed in version 1.14.8-1+deb9u1.
• Issued DLA 2525-1, fixing CVE-2018-19840, CVE-2018-19841, CVE-2019-11498, CVE-2019-1010315, CVE-2019-1010317, CVE-2019-1010319, and CVE-2020-35738, for wavpack.
  For Debian 9 Stretch, these problems have been fixed in version 5.0.0-2+deb9u3.
• Issued DLA 2526-1, fixing CVE-2020-26298, for ruby-redcarpet.
  For Debian 9 Stretch, these problems have been fixed in version 3.3.4-2+deb9u1.
• Prepared DSA 4831-1, fixing CVE-2020-26298, for ruby-redcarpet. For Debian 10 Buster, these problems have been fixed in version 3.4.0-4+deb10u1. The announcement was released by the Security Team.
• Issued DLA 2528-1, fixing CVE-2021-3185, for gst-plugins-bad1.0.
  For Debian 9 Stretch, these problems have been fixed in version 1.10.4-1+deb9u1.
• Issued DLA 2529-1, fixing CVE-2021-3181, for mutt.
  For Debian 9 Stretch, these problems have been fixed in version 1.7.2-1+deb9u5.
• Issued DLA 2531-1), fixing CVE-2020-28473, for python-bottle.
  For Debian 9 Stretch, these problems have been fixed in version 0.12.13-1+deb9u1.
• Released buster-pu update, fixing CVE-2020-28473, for python-bottle. For Debian 10 Buster, these problems have been fixed in version 0.12.15-2+deb10u1.

ELTS CVE Fixes and Announcements:

• Issued ELA 344-1, fixing CVE-2020-27350, for apt.
  For Debian 8 Jessie, these problems have been fixed in version 1.0.9.8.7.
• Issued ELA 346-1, fixing CVE-2016-10169, CVE-2018-19840, CVE-2019-1010319, and CVE-2020-35738, for wavpack.
  For Debian 8 Jessie, these problems have been fixed in version 4.70.0-1+deb8u1.
• Issued ELA 347-1, fixing CVE-2020-26298, for ruby-redcarpet.
  For Debian 8 Jessie, these problems have been fixed in version 3.1.2-1+deb8u1.
• Issued ELA 348-1, fixing CVE-2021-3185, for gst-plugins-bad1.0.
  For Debian 8 Jessie, these problems have been fixed in version 1.4.4-2.1+deb8u3.
• Issued ELA 349-1, fixing CVE-2021-3181, for mutt.
  For Debian 8 Jessie, these problems have been fixed in version 1.5.23-3+deb8u5.
• Issued ELA 350-1, fixing CVE-2020-28473, for python-bottle.

Other (E)LTS Work:

• Front-desk duty from 28-12 until 03-01 and from 25-01 until 31-01 for both LTS and ELTS.
• Triaged dropbear, gst-plugins-bad1.0, phpmyadmin, qemu, firefox-esr, thunderbird, openldap, libdatetime-timezone-perl, tzdata, jasper, ckeditor, liblivemedia, wavpack, and ruby-redcarpet.
• Marked CVE-2019-12953/dropbear as postponed for jessie.
• Marked CVE-2019-12953/dropbear as postponed for stretch.
• Marked CVE-2018-19841/wavpack as not-affected for jessie.
• Marked CVE-2019-1010315/wavpack as not-affected for jessie.
• Marked CVE-2019-1010317/wavpack as not-affected for jessie.
• Marked CVE-2021-21252/phpmyadmin as no-dsa for stretch.
• Marked CVE-2021-20196/qemu as postponed for stretch.
• Marked CVE-2021-21252/phpmyadmin as no-dsa for jessie.
• Marked CVE-2021-20196/qemu as postponed for jessie.
• Marked CVE-2020-11947/qemu as postponed for jessie.
• Marked CVE-2021-3326/glibc as no-dsa for jessie.
• Marked CVE-2021-3326/glibc as no-dsa for stretch.
• Marked CVE-2020-35517/qemu as not-affected instead of postponed for jessie.
• Marked CVE-2021-2627 1,2 /ckeditor as postponed for jessie.
• Marked CVE-2020-24027/liblivemedia as no-dsa for stretch.
• Marked CVE-2021-2627 1,2 /ckeditor as postponed for stretch.
• Auto EOL ed csync2, firefox-esr, linux, thunderbird, collabtive, activemq, and xen for jessie.
• Got my first ever CVE assigned - CVE-2021-3181 for mutt. Weeeehooooo! \o/
• Attended the monthly LTS meeting. Logs here.
• General discussion on LTS private and public mailing list.

---

Interesting Bits!

• This January, on 23rd and 24th, we had Mini DebConf India 2021 online.
  I had a talk as well, titled, Why Point Releases are important and how you can help prepare them?". It was a fun and a very short talk, where I just list out the reasons and ways to help in the preparation of point releases . I did some experimentation with this talk, figuring out what works for the audience and what doesn t and where can I improve for the next time I talk about this topic! \o/
  You can listen to the talk here and let me know if you have any feedback! Anyway, the conference lasted for 2 days and I also did some volunteering (talk director, talk miester) in Hindi and English, both! It was all so fun and new. Anyway, here s the picture we took:
• In another exciting news, I got my first CVE assigned!!! \o/
  No, it is not something that I found, it was discovered by Tavis Ormandy. I just assigned this a CVE ID, CVE-2021-3181.
  This is my first, so I am very excited about this! ^_^
• Besides, there s something more that is in the pipelines. Can t talk about it now, shh. But hopefully very sooooooon!

---

Other $things! \o/ This month was tiresome, with most of the time being spent on the Debian stuff, I did very little work outside it, really. The issues and patches that I sent are:

• Issue #700 for redcarpet, asking for a reproducer for CVE-2020-26298 and some additional patch related queries.
• Issue #7 for in-parallel, asking them to not use relative paths for tests.
• Issue #8 for in-parallel, reporting a test failure for the library.
• Issue #2 for rake-ant, asking them to bump their dependencies to a newer version.
• PR #3 for rake-ant, bumping the dependencies to a newer version, fixing the above issue, heh.
• Issue #4 for rake-ant, requesting to drop git from their gemspec.
• PR #5 for rake-ant, dropping git from gemspec, fixing the above issue, heh.
• Issue #95 for WavPack, asking for a review of past security vulnerabilites wrt v4.70.0.
• Reviewed PR #128 for ruby-openid, addressing the past regression with CVE fix merge.
• Reviewed PR #63 for cocoapods-acknowledgements, updating redcarpet to v3.5.1, as a safety measure due to recently discovered vulnerability.
• Issue #1331 for bottle, asking for relevant commits for CVE-2020-28473 and clarifying other things.
• Issue #5 for em-redis, reporting test failures on IPv6-only build machines.
• Issue #939 for eventmachine, reporting test failures for em-redis on IPv6-only build machines.

---

Until next time.
:wq for today.

Welcome to the October 2020 report from the Reproducible Builds project. In our monthly reports, we outline the major things that we have been up to over the past month. As a brief reminder, the motivation behind the Reproducible Builds effort is to ensure flaws have not been introduced in the binaries we install on our systems. If you are interested in contributing to the project, please visit our main website.

General On Saturday 10th October, Morten Linderud gave a talk at Arch Conf Online 2020 on The State of Reproducible Builds in Arch. The video should be available later this month, but as a teaser:

The previous year has seen great progress in Arch Linux to get reproducible builds in the hands of the users and developers. In this talk we will explore the current tooling that allows users to reproduce packages, the rebuilder software that has been written to check packages and the current issues in this space.

During the Reproducible Builds summit in Marrakesh in 2019, developers from the GNU Guix, NixOS and Debian distributions were able to produce a bit-for-bit identical GNU Mes binary despite using three different versions of GCC. Since this summit, additional work resulted in a bit-for-bit identical Mes binary using tcc, and last month a fuller update was posted to this effect by the individuals involved. This month, however, David Wheeler updated his extensive page on Fully Countering Trusting Trust through Diverse Double-Compiling, remarking that:

GNU Mes rebuild is definitely an application of [Diverse Double-Compiling]. [..] This is an awesome application of DDC, and I believe it s the first publicly acknowledged use of DDC on a binary

There was a small, followup discussion on our mailing list. In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update. This month, the Reproducible Builds project restarted our IRC meetings, managing to convene twice: the first time on October 12th (summary & logs), and later on the 26th (logs). As mentioned in previous reports, due to the unprecedented events throughout 2020, there will be no in-person summit event this year. On our mailing list this month El as Alejandro posted a request for help with a local configuration

Debian-related work In August, Lucas Nussbaum performed an archive-wide rebuild of packages to test enabling the reproducible=+fixfilepath Debian build flag by default. Enabling this fixfilepath feature will likely fix reproducibility issues in an estimated 500-700 packages. However, this month Vagrant Cascadian posted to the debian-devel mailing list:

It would be great to see the reproducible=+fixfilepath feature enabled by default in dpkg-buildflags, and we would like to proceed forward with this soon unless we hear any major concerns or other outstanding issues. [ ] We would like to move forward with this change soon, so please raise any concerns or issues not covered already.

Debian Developer Stuart Prescott has been improving python-debian, a Python library that is used to parse Debian-specific files such as changelogs, .dscs, etc. In particular, Stuart is working on adding support for .buildinfo files used for recording reproducibility-related build metadata:

This can mostly be a very thin layer around the existing Deb822 types, using the existing Changes code for the file listings, the existing PkgRelations code for the package listing and gpg_* functions for signature handling.

A total of 159 Debian packages were categorised, 69 had their categorisation updated, and 33 had their classification removed this month, adding to our knowledge about identified issues. As part of this, Chris Lamb identified and classified two new issues: build_path_captured_in_emacs_el_file and rollup_embeds_build_path.

Software development This month, we tried to fix a large number of currently-unreproducible packages, including:

• Bernhard M. Wiedemann:
  • go (version 1.15.3 has improved reproducibility over 1.14)
  • goxel (sort SCons-related filesystem ordering issue)
  • lal (rework an old date-related patch)
  • lalmetaio (date)
  • libsemigroups (build failure in single-CPU mode)
  • memcached (build failure in 2025 due to expired SSL certificate)
  • octant (SUSE-specific date issue)
  • openmpi4 (date-related problem, revive old patch)
  • sbcl (datetime and hostname issue)
  • selinux-policy/policycoreutils (date-related issue in timezone)
• Chris Lamb:
  • #970383 filed against evince (forwarded upstream).
  • #971527 filed against libsass-python (forwarded upstream).
  • #972077 filed against pitivi.
  • #972078 filed against sound-juicer.
  • #972147 filed against pcbasic.
  • #972336 filed against ora2pg (forwarded upstream).
  • #972378 filed against fckit.
  • #972493 filed against gita.
  • #972494 filed against libgrokj2k.
  • #972496 filed against softether-vpn.
  • #972559 filed against perl.
  • #972561 filed against ruby-appraiser.
  • #972562 filed against gmerlin-avdecoder.
  • #972631 filed against node-proxy.
  • #972668 filed against yard.
  • #972861 filed against emacs.
  • #972930 filed against netcdf-parallel.
  • #965255 re-opened with new patch dh-fortran-mod.

Bernhard M. Wiedemann also reported three issues against bison, ibus and postgresql12.

Tools diffoscope is our in-depth and content-aware diff utility. Not only could you locate and diagnose reproducibility issues, it provides human-readable diffs of all kinds too. This month, Chris Lamb uploaded version 161 to Debian (later backported by Mattia Rizzolo), as well as made the following changes:

• Move test_ocaml to the assert_diff helper. [ ]
• Update tests to support OCaml version 4.11.1. Thanks to Sebastian Ramacher for the report. (#972518)
• Bump minimum version of the Black source code formatter to 20.8b1. (#972518)

In addition, Jean-Romain Garnier temporarily updated the dependency on radare2 to ensure our test pipelines continue to work [ ], and for the GNU Guix distribution Vagrant Cascadian diffoscope to version 161 [ ]. In related development, trydiffoscope is the web-based version of diffoscope. This month, Chris Lamb made the following changes:

• Mark a --help-only test as being a superficial test. (#971506)
• Add a real, albeit flaky, test that interacts with the try.diffoscope.org service. [ ]
• Bump debhelper compatibility level to 13 [ ] and bump Standards-Version to 4.5.0 [ ].

Lastly, disorderfs version 0.5.10-2 was uploaded to Debian unstable by Holger Levsen, which enabled security hardening via DEB_BUILD_MAINT_OPTIONS [ ] and dropped debian/disorderfs.lintian-overrides [ ].

Website and documentation This month, a number of updates to the main Reproducible Builds website and related documentation were made by Chris Lamb:

• Add a citation link to the academic article regarding dettrace [ ], and added yet another supply-chain security attack publication [ ].
• Reformatted the Jekyll s Liquid templating language and CSS formatting to be consistent [ ] as well as expand a number of tab characters [ ].
• Used relative_url to fix missing translation icon on various pages. [ ]
• Published two announcement blog posts regarding the restarting of our IRC meetings. [ ][ ]
• Added an explicit note regarding the lack of an in-person summit in 2020 to our events page. [ ]

Testing framework The Reproducible Builds project operates a Jenkins-based testing framework that powers tests.reproducible-builds.org. This month, Holger Levsen made the following changes:

• Debian-related changes:
  • Refactor and improve the Debian dashboard. [ ][ ][ ]
  • Track bugs which are usertagged as filesystem , fixfilepath , etc.. [ ][ ][ ]
  • Make a number of changes to package index pages. [ ][ ][ ]
• System health checks:
  • Relax disk space warning levels. [ ]
  • Specifically detect build failures reported by dpkg-buildpackage. [ ]
  • Fix a regular expression to detect outdated package sets. [ ]
  • Detect Lintian issues in diffoscope. [ ]

• Misc:
  • Make a number of updates to reflect that our sponsor Profitbricks has renamed itself to IONOS. [ ][ ][ ][ ]
  • Run a F-Droid maintenance routine twice a month to utilise its cleanup features. [ ]
  • Fix the target name in OpenWrt builds to ath79 from ath97. [ ]
  • Add a missing Postfix configuration for a node. [ ]
  • Temporarily disable Arch Linux builds until a core node is back. [ ]
  • Make a number of changes to our thanks page. [ ][ ][ ]

Build node maintenance was performed by both Holger Levsen [ ][ ] and Vagrant Cascadian [ ][ ][ ], Vagrant Cascadian also updated the page listing the variations made when testing to reflect changes for in build paths [ ] and Hans-Christoph Steiner made a number of changes for F-Droid, the free software app repository for Android devices, including:

• Do not fail reproducibility jobs when their cleanup tasks fail. [ ]
• Skip libvirt-related sudo command if we are not actually running libvirt. [ ]
• Use direct URLs in order to eliminate a useless HTTP redirect. [ ]

---

If you are interested in contributing to the Reproducible Builds project, please visit the Contribute page on our website. However, you can also get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mailing list: rb-general@lists.reproducible-builds.org
• Social media: @ReproBuilds, @reproducible_builds@fosstodon.org & Reddit

Welcome to the August 2020 report from the Reproducible Builds project. In our monthly reports, we summarise the things that we have been up to over the past month. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced from the original free software source code to the pre-compiled binaries we install on our systems. If you re interested in contributing to the project, please visit our main website.

---

This month, Jennifer Helsby launched a new reproduciblewheels.com website to address the lack of reproducibility of Python wheels. To quote Jennifer s accompanying explanatory blog post:

One hiccup we ve encountered in SecureDrop development is that not all Python wheels can be built reproducibly. We ship multiple (Python) projects in Debian packages, with Python dependencies included in those packages as wheels. In order for our Debian packages to be reproducible, we need that wheel build process to also be reproducible

Parallel to this, transparencylog.com was also launched, a service that verifies the contents of URLs against a publicly recorded cryptographic log. It keeps an append-only log of the cryptographic digests of all URLs it has seen. (GitHub repo) On 18th September, Bernhard M. Wiedemann will give a presentation in German, titled Wie reproducible builds Software sicherer machen ( How reproducible builds make software more secure ) at the Internet Security Digital Days 2020 conference.

Reproducible builds at DebConf20 There were a number of talks at the recent online-only DebConf20 conference on the topic of reproducible builds. Holger gave a talk titled Reproducing Bullseye in practice , focusing on independently verifying that the binaries distributed from ftp.debian.org are made from their claimed sources. It also served as a general update on the status of reproducible builds within Debian. The video (145 MB) and slides are available. There were also a number of other talks that involved Reproducible Builds too. For example, the Malayalam language mini-conference had a talk titled , ? ( I want to join Debian, what should I do? ) presented by Praveen Arimbrathodiyil, the Clojure Packaging Team BoF session led by Elana Hashman, as well as Where is Salsa CI right now? that was on the topic of Salsa, the collaborative development server that Debian uses to provide the necessary tools for package maintainers, packaging teams and so on. Jonathan Bustillos (Jathan) also gave a talk in Spanish titled Un camino verificable desde el origen hasta el binario ( A verifiable path from source to binary ). (Video, 88MB)

Development work After many years of development work, the compiler for the Rust programming language now generates reproducible binary code. This generated some general discussion on Reddit on the topic of reproducibility in general. Paul Spooren posted a request for comments to OpenWrt s openwrt-devel mailing list asking for clarification on when to raise the PKG_RELEASE identifier of a package. This is needed in order to successfully perform rebuilds in a reproducible builds context. In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update. Chris Lamb provided some comments and pointers on an upstream issue regarding the reproducibility of a Snap / SquashFS archive file. [ ]

Debian Holger Levsen identified that a large number of Debian .buildinfo build certificates have been tainted on the official Debian build servers, as these environments have files underneath the /usr/local/sbin directory [ ]. He also filed against bug for debrebuild after spotting that it can fail to download packages from snapshot.debian.org [ ]. This month, several issues were uncovered (or assisted) due to the efforts of reproducible builds. For instance, Debian bug #968710 was filed by Simon McVittie, which describes a problem with detached debug symbol files (required to generate a traceback) that is unlikely to have been discovered without reproducible builds. In addition, Jelmer Vernooij called attention that the new Debian Janitor tool is using the property of reproducibility (as well as diffoscope when applying archive-wide changes to Debian:

New merge proposals also include a link to the diffoscope diff between a vanilla build and the build with changes. Unfortunately these can be a bit noisy for packages that are not reproducible yet, due to the difference in build environment between the two builds. [ ]

56 reviews of Debian packages were added, 38 were updated and 24 were removed this month adding to our knowledge about identified issues. Specifically, Chris Lamb added and categorised the nondeterministic_version_generated_by_python_param and the lessc_nondeterministic_keys toolchain issues. [ ][ ] Holger Levsen sponsored Lukas Puehringer s upload of the python-securesystemslib pacage, which is a dependency of in-toto, a framework to secure the integrity of software supply chains. [ ] Lastly, Chris Lamb further refined his merge request against the debian-installer component to allow all arguments from sources.list files (such as [check-valid-until=no]) in order that we can test the reproducibility of the installer images on the Reproducible Builds own testing infrastructure and sent a ping to the team that maintains that code.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of these patches, including:

• Bernhard M. Wiedemann:
  • asymptote (shell/Perl date)
  • getfem (embeds datetime and user, submitted via email)
  • getdp (hostname and user)
  • getdp (user)
  • guix (disable parallelism)
  • httpcomponents-client (Java documentation generator readdir order)
  • kuberlr (date)
  • lal (date and time issue, submitted via email)
  • libmesh (host)
  • OBS (discuss how to track old build prjconf metadata in buildinfo)
  • openblas (disable CPU detection)
  • openfoam-selector (date)
  • perl (toolchain, date)
  • python-blosc (CPU detection)
  • python-eventlet (fails to build far in the future)
  • rna-star (date and hostname)
  • trilinos (date)
  • xz/b4 (workaround CPU count influencing output, reported upstream)
• Benjamin Hof:
  • flit
• Chris Lamb:
  • #966657 filed against json-c (forwarded upstream).
  • #967238 filed against nmh.
  • #968045 filed against golang-gonum-v1-plot.
  • #968183 filed against chirp.
  • #968185 filed against pixelmed-codec.
  • #968187 filed against debhelper.
  • #968189 filed against muroar.
  • #968278 filed against serd.
  • #968344 filed against pencil2d.
  • #968557 filed against tpot.
  • #968700 filed against evolution.
  • #969320 filed against aflplusplus.
• Herv Boutemy:
  • plexus-archiver (timezone/DST issue)
• Vagrant Cascadian:
  • #968627 filed against libjpeg-turbo.
  • #968641 filed against jack-audio-connection-kit.
  • #968652 filed against glusterfs.

diffoscope diffoscope is our in-depth and content-aware diff utility that can not only locate and diagnose reproducibility issues, it provides human-readable diffs of all kinds. In August, Chris Lamb made the following changes to diffoscope, including preparing and uploading versions 155, 156, 157 and 158 to Debian:

• New features:
  • Support extracting data of PGP signed data. (#214)
  • Try files named .pgp against pgpdump(1) to determine whether they are Pretty Good Privacy (PGP) files. (#211)
  • Support multiple options for all file extension matching. [ ]
• Bug fixes:
  • Don t raise an exception when we encounter XML files with <!ENTITY> declarations inside the Document Type Definition (DTD), or when a DTD or entity references an external resource. (#212)
  • pgpdump(1) can successfully parse some binary files, so check that the parsed output contains something sensible before accepting it. [ ]
  • Temporarily drop gnumeric from the Debian build-dependencies as it has been removed from the testing distribution. (#968742)
  • Correctly use fallback_recognises to prevent matching .xsb binary XML files.
  • Correct identify signed PGP files as file(1) returns data . (#211)
• Logging improvements:
  • Emit a message when ppudump version does not match our file header. [ ]
  • Don t use Python s repr(object) output in Calling external command messages. [ ]
  • Include the filename in the not identified by any comparator message. [ ]
• Codebase improvements:
  • Bump Python requirement from 3.6 to 3.7. Most distributions are either shipping with Python 3.5 or 3.7, so supporting 3.6 is not only somewhat unnecessary but also cumbersome to test locally. [ ]
  • Drop some unused imports [ ], drop an unnecessary dictionary comprehensions [ ] and some unnecessary control flow [ ].
  • Correct typo of output in a comment. [ ]
• Release process:
  • Move generation of debian/tests/control to an external script. [ ]
  • Add some URLs for the site that will appear on PyPI.org. [ ]
  • Update author and author email in setup.py for PyPI.org and similar. [ ]
• Testsuite improvements:
  • Update PPU tests for compatibility with Free Pascal versions 3.2.0 or greater. (#968124)
  • Mark that our identification test for .ppu files requires ppudump version 3.2.0 or higher. [ ]
  • Add an assert_diff helper that loads and compares a fixture output. [ ][ ][ ][ ]
• Misc:
  • Duplicate docker instructions in the Get diffoscope section of the diffoscope website. [ ]

In addition, Mattia Rizzolo documented in setup.py that diffoscope works with Python version 3.8 [ ] and Frazer Clews applied some Pylint suggestions [ ] and removed some deprecated methods [ ].

Website This month, Chris Lamb updated the main Reproducible Builds website and documentation to:

• Clarify & fix a few entries on the who page [ ][ ] and ensure that images do not get to large on some viewports [ ].
• Clarify use of a pronoun re. Conservancy. [ ]
• Use View all our monthly reports over View all monthly reports . [ ]
• Move a is a suffix out of the link target on the SOURCE_DATE_EPOCH age. [ ]

In addition, Javier Jard n added the freedesktop-sdk project [ ] and Kushal Das added SecureDrop project [ ] to our projects page. Lastly, Michael P hn added internationalisation and translation support with help from Hans-Christoph Steiner [ ].

Testing framework The Reproducible Builds project operate a Jenkins-based testing framework to power tests.reproducible-builds.org. This month, Holger Levsen made the following changes:

• System health checks:
  • Improve explanation how the status and scores are calculated. [ ][ ]
  • Update and condense view of detected issues. [ ][ ]
  • Query the canonical configuration file to determine whether a job is disabled instead of duplicating/hardcoding this. [ ]
  • Detect several problems when updating the status of reporting-oriented metapackage sets. [ ]
  • Detect when diffoscope is not installable [ ] and failures in DNS resolution [ ].
• Debian:
  • Update the URL to the Debian security team bug tracker s Git repository. [ ]
  • Reschedule the unstable and bullseye distributions often for the arm64 architecture. [ ]
  • Schedule buster less often for armhf. [ ][ ][ ]
  • Force the build of certain packages in the work-in-progress package rebuilder. [ ][ ]
  • Only update the stretch and buster base build images when necessary. [ ]
• Other distributions:
  • For F-Droid, trigger jobs by commits, not by a timer. [ ]
  • Disable the Archlinux HTML page generation job as it has never worked. [ ]
  • Disable the alternative OpenWrt rebuilder jobs. [ ]
• Misc;
  • Improve monitoring, such as number of mounts, disk, memory, etc.. [ ][ ][ ][ ]
  • Install the ruby-jekyll-polyglot package to needed for the recently-added internationalisation and translation support on the Reproducible Builds website. [ ]
  • Update link to report potential issues. [ ][ ]

Many other changes were made too, including:

• Chris Lamb:
  • Use <pre> HTML tags when dumping fixed-width debugging data in the self-serve package scheduler. [ ]
• Mattia Rizzolo:
  • For Alpine and ArchLinux, make the cleanup routines in the event of an error more robust. [ ]
  • Update the sudo configuration to permit Jenkins itself to unmount more directories. [ ]
  • Setup automatic renewal of our Let s Encrypt certificates for all domains served by us (including jenkins.debian.net, www.reproducible-builds.org, diffoscope.org, buildinfos.debian.net, etc.). [ ][ ][ ][ ][ ]
• Vagrant Cascadian:
  • Mark that the u-boot Universal Boot Loader should not build architecture independent packages on the arm64 architecture anymore. [ ]

Finally, build node maintenance was performed by Holger Levsen [ ], Mattia Rizzolo [ ][ ] and Vagrant Cascadian [ ][ ][ ][ ]

---

Mailing list On our mailing list this month, Leo Wandersleb sent a message to the list after he was wondering how to expand his WalletScrutiny.com project (which aims to improve the security of Bitcoin wallets) from Android wallets to also monitor Linux wallets as well:

If you think you know how to spread the word about reproducibility in the context of Bitcoin wallets through WalletScrutiny, your contributions are highly welcome on this PR [ ]

Julien Lepiller posted to the list linking to a blog post by Tavis Ormandy titled You don t need reproducible builds. Morten Linderud (foxboron) responded with a clear rebuttal that Tavis was only considering the narrow use-case of proprietary vendors and closed-source software. He additionally noted that the criticism that reproducible builds cannot prevent against backdoors being deliberately introduced into the upstream source ( bugdoors ) are decidedly (and deliberately) outside the scope of reproducible builds to begin with. Chris Lamb included the Reproducible Builds mailing list in a wider discussion regarding a tentative proposal to include .buildinfo files in .deb packages, adding his remarks regarding requiring a custom tool in order to determine whether generated build artifacts are identical in a reproducible context. [ ] Jonathan Bustillos (Jathan) posted a quick email to the list requesting whether there was a list of To do tasks in Reproducible Builds. Lastly, Chris Lamb responded at length to a query regarding the status of reproducible builds for Debian ISO or installation images. He noted that most of the technical work has been performed but there are at least four issues until they can be generally advertised as such . He pointed that the privacy-oriented Tails operation system, which is based directly on Debian, has had reproducible builds for a number of years now. [ ]

---

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Twitter: @ReproBuilds
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org

Dealing with the void during MiniDebConf Online #1 Between 28 and 31 May this year, we set out to create our first ever online MiniDebConf for Debian. Many people have been meaning to do something similar for a long time, but it just didn t work out yet. With many of us being in lock down due to COVID-19, and with the strong possibility looming that DebConf20 might have had to become an online event, we rushed towards organising the first ever Online MiniDebConf and put together some form of usable video stack for it. I could go into all kinds of details on the above, but this post is about a bug that lead to a pretty nifty feature for DebConf20. The tool that we use to capture Jitsi calls is called Jibri (Jitsi Broadcasting Infrustructure). It had a bug (well, bug for us, but it s an upstream feature) where Jibri would hang up after 30s of complete silence, because it would assume that the call has ended and that the worker can be freed up again. This would result in the stream being ended at the end of every talk, so before the next talk, someone would have to remember to press play again in their media player or on the video player on the stream page. Hrmph. Easy solution on the morning that the conference starts? I was testing a Debian Live image the night before in a KVM and thought that I might as well just start a Jitsi call from there and keep a steady stream of silence so that Jibri doesn t hang up. It worked! But the black screen and silence on stream was a bit eery. Because this event was so experimental in nature, and because we were on such an incredibly tight timeline, we opted not to seek sponsors for this event, so there was no sponsors loop that we d usually stream during a DebConf event. Then I thought Ah! I could just show the schedule! .

The stream looked bright and colourful (and was even useful!) and Jitsi/Jibri didn t die. I thought my work was done. As usual, little did I know how untrue that was. The silence was slightly disturbing after the talks, and people asked for some music. Playing music on my VM and capturing the desktop audio in to Jitsi was just a few pulseaudio settings away, so I spent two minutes finding some freely licensed tracks that sounded ok enough to just start playing on the stream. I came across mini-albums by Captive Portal and Cinema Noir, During the course of the MiniDebConf Online I even started enjoying those. Someone also pointed out that it would be really nice to have a UTC clock on the stream. I couldn t find a nice clock in a hurry so I just added a tmux clock in the meantime while we deal with the real-time torrent of issues that usually happens when organising events like this.

Speaking of issues, during our very first talk of the last day, our speaker had a power cut during the talk and abruptly dropped off. Oops! So, since I had a screenshare open from the VM to the stream, I thought I d just pop in a quick message in a text editor to let people know that we re aware of it and trying to figure out what s going on.

In the end, MiniDebConf Online worked out all right. Besides the power cut for our one speaker, and another who had a laptop that was way too under-powered to deal with video, everything worked out very well. Even the issues we had weren t show-stoppers and we managed to work around them.

DebConf20 Moves Online For DebConf, we usually show a sponsors loop in between sessions. It s great that we give our sponsors visibility here, but in reality people see the sponsors loop and think Talk over! and then they look away. It s also completely silent and doesn t provide any additional useful information. I was wondering how I could take our lessons from MDCO#1 and integrate our new tricks with the sponsors loop. That is, add the schedule, time, some space to type announcements on the screen and also add some loopable music to it. I used OBS before in making my videos, and like the flexibility it provides when working with scenes and sources. A scene is what you would think of as a screen or a document with its own collection of sources or elements. For example, a scene might contain sources such as a logo, clock, video, image, etc. A scene can also contain another scene. This is useful if you want to contain a banner or play some background music that is shared between scenes.

The above screenshots illustrate some basics of scenes and sources. First with just the DC20 banner, and then that used embedded in another scene. For MDCO#1, I copied and pasted the schedule into a LibreOffice Impress slide that was displayed on the stream. Having to do this for all 7 days of DebConf, plus dealing with scheduling changes would be daunting. So, I started to look in to generating some schedule slides programmatically. Stefano then pointed me to the Happening Now page on the DebConf website, where the current schedule block is displayed. So all I would need to do in OBS was to display a web page. Nice! Unfortunately the OBS in Debian doesn t have the ability to display web pages out of the box (we need to figure out CEF in Debian), but fortunately someone provides a pre-compiled version of the plugin called Linux Browser that works just fine. This allowed me to easily add the schedule page in its own scene. Being able to display a web page solved another problem. I wasn t fond of having to type / manage the announcements in OBS. It would either be a bit prone to user error, and if you want to edit the text while the loop is running, you d have to disrupt the loop, go to the foreground scene, and edit the text before resuming the loop. That s a bit icky. Then I thought that we could probably just get that from a web page instead. We could host some nice html snippet in a repository in salsa, and then anyone could easily commit an MR to update the announcement. But then I went a step further, use an etherpad! Then anyone in the orga team can quickly update the announcement and it would be instantly changed on the stream. Nice! So that small section of announcement text on the screen is actually a whole web browser with an added OBS filter to crop away all the pieces we don t want. Overkill? Sure, but it gave us a decent enough solution that worked in time for the start of DebConf. Also, being able to type directly on to the loop screen works out great especially in an emergency. Oh, and uhm the clock is also a website rendered in its own web browser :-P

So, I had the ability to make scenes, add elements and add all the minimal elements I wanted in there. Great! But now I had to figure out how to switch scenes automatically. It s probably worth mentioning that I only found some time to really dig into this right before DebConf started, so with all of this I was scrambling to find things that would work without too many bugs while also still being practical. Now I needed the ability to switch between the scenes automatically / programmatically. I had never done this in OBS before. I know it has some API because there are Android apps that you can use to control OBS with from your phone. I discovered that it had an automatic scene switcher, but it s very basic. It can only switch based on active window, which can be useful in some cases, but since we won t have any windows open other than OBS, this tool was basically pointless.

After some quick searches, I found a plugin called Advanced Scene Switcher. This plugin can do a lot more, but has some weird UI choices, and is really meant for gamers and other types of professional streamers to help them automate their work flow and doesn t seem at all meant to be used for a continuous loop, but, it worked, and I could make it do something that will work for us during the DebConf. I had a chicken and egg problem because I had to figure out a programming flow, but didn t really have any content to work with, or an idea of all the content that we would eventually have. I ve been toying with the idea in my mind and had some idea that we could add fun facts, postcards (an image with some text), time now in different timezones, Debian news (maybe procured by the press team), cards that contain the longer announcements that was sent to debconf-announce, perhaps a shout out or two and some photos from previous DebConfs like the group photos. I knew that I wouldn t be able to build anything substantial by the time DebConf starts, but adding content to OBS in between talks is relatively easy, so we could keep on building on it during DebConf. Nattie provided the first shout out, and I made 2 video loops with the DC18/19 pictures and also two Did you know cards. So the flow I ended up with was: Sponsors -> Happening Now -> Random video (which would be any of those clips) -> Back to sponsors. This ended up working pretty well for quite a while. With the first batch of videos the sponsor loop would come up on average about every 2 minutes, but as much shorter clips like shout outs started to come in faster and faster, it made sense to play a few 2-3 shout-outs before going back to sponsors. So here is a very brief guide on how I set up the sequencing in Advanced Scene Switcher.

If no condition was met, a video would play from the Random tab.

Then in the Random tab, I added the scenes that were part of the random mix. Annoyingly, you have to specify how long it should play for. If you don t, the no condition thingy is triggered and another video is selected. The time is also the length of the video minus one second, because

You can t just say that a random video should return back to a certain scene, you have to specify that in the sequence tab for each video. Why after 1 second? Because, at least in my early tests, and I didn t circle back to this, it seems like 0s can randomly either mean instantly, or never. Yes, this ended up being a bit confusing and tedious, and considering the late hours I worked on this, I m surprised that I didn t manage to screw it up completely at any point. I also suspected that threads would eventually happen. That is, when people create video replies to other videos. We had 3 threads in total. There was a backups thread, beverage thread and an impersonation thread. The arrow in the screenshot above points to the backups thread. I know it doesn t look that complicated, but it was initially somewhat confusing to set up and make sense out of it.

For the next event, the Advanced Scene Switcher might just get some more taming, or even be replaced entirely. There are ways to drive OBS by API, and even the Advanced Scene Switcher tool can be driven externally to some degree, but I think we definitely want to replace it by the next full DebConf. We had the problem that when a talk ended, we would return to the loop in the middle of a clip, which felt very unnatural and sometimes even confusing. So Stefano helped me with a helper script that could read the socket from Vocto, which I used to write either Loop or Standby to a file, and then the scene switcher would watch that file and keep the sponsors loop ready for start while the talks play. Why not just switch to sponsors when the talk ends? Well, the little bit of delay in switching would mean that you would see a tiny bit of loop every time before switching to sponsors. This is also why we didn t have any loop for the ad-hoc track (that would have probably needed another OBS instance, we ll look more into solutions for this for the future).

Then for all the clips. There were over 50 of them. All of them edited by hand in kdenlive. I removed any hard clicks, tried to improve audibility, remove some sections at the beginning and the end that seemed extra and added some music that would reduce in volume when someone speaks. In the beginning, I had lots of fun with choosing music for the clips. Towards the end, I had to rush them through and just chose the same tune whether it made sense or not. For comparison of what a difference the music can make, compare the original and adapted version for Valhalla s clip above, or this original and adapted video from urbec. This part was a lot more fun than dealing with the video sequencer, but I also want to automate it a bit. When I can fully drive OBS from Python I ll likely instead want to show those cards and control music volume from Python (what could possibly go wrong ). The loopy name happened when I requested an @debconf.org alias for this. I was initially just thinking about loop@debconf.org but since I wanted to make it clear that the purpose of this loop is also to have some fun, I opted for loopy instead:

I was really surprised by how people took to loopy. I hoped it would be good and that it would have somewhat positive feedback, but the positive feedback was just immense. The idea was that people typically saw it in between talks. But a few people told me they kept it playing after the last talk of the day to watch it in the background. Some asked for the music because they want to keep listening to it while working (and even for jogging!?). Some people also asked for recordings of the loop because they want to keep it for after DebConf. The shoutouts idea proved to be very popular. Overall, I m very glad that people enjoyed it and I think it s safe to say that loopy will be back for the next event.

Also throughout this experiment Loopy Loop turned into yet another DebConf mascot. We gain one about every DebConf, some by accident and some on purpose. This one was not quite on purpose. I meant to make an image for it for salsa, and started with an infinite loop symbol. That s a loop, but by just adding two more solid circles to it, it looks like googly eyes, now it s a proper loopy loop! I like the progress we ve made on this, but there s still a long way to go, and the ideas keep heaping up. The next event is quite soon (MDCO#2 at the end of November, and it seems that 3 other MiniDebConf events may also be planned), but over the next few events there will likely be significantly better graphics/artwork, better sequencing, better flow and more layout options. I hope to gain some additional members in the team to deal with incoming requests during DebConf. It was quite hectic this time! The new OBS also has a scripting host that supports Python, so I should be able to do some nice things even within OBS without having to drive it externally (like, display a clock without starting a web browser).

The Loopy Loop Music The two mini albums that mostly played during the first few days were just a copy and paste from the MDCO#1 music, which was:

• Captive Portal What is
• Cinema Noir Children kill ants

For shoutout tracks, that were later used in the loop too (because it became a bit monotonous), most of the tracks came from freepd.com:

• Foam Rubber (used in Nattie s and Urbec s shoutouts, and other places)
• Metaltania (used in Luna s shoutout)
• Flying Jack (used in Pollo s epic videoteam call for help)
• Study and relax (used in Rhonda s poetry night announcement)
• City sunshine (used in gregoa s coffee shoutout and then in several more after)
• Other tracks played in the loop from freepd.com included Brothers Unite, Ice and Snow, Parhelion, Shining stars and The Lagoon.

I have much more things to say about DebConf20, but I ll keep that for another post, and hopefully we can get all the other video stuff in a post from the video team, because I think there s been some real good work done for this DebConf. Also thanks to Infomaniak who was not only a platinum sponsor for this DebConf, but they also provided us with plenty of computing power to run all the video stuff on. Thanks again!

The DebConf 20 song - a sea shanty - to the tune of "Fathom the bowl"

Here's to DebConf 20, the brightest and best
Now it's this year's orga team getting no rest
We're not met in Haifa - it's all doom and gloom
And I'm sat like a lifer here trapped in my room

I'm sat in my room, it's all doom and gloom
And I'm sat at my keyboard here trapped in my room

Now there's IRC rooms and there's jitsi and all
But no fun conversations as we meet in the hall
No hugs for old friends, no shared wine and cheese
Just shared indigestion as we take our ease

I'm sat in my room, it's all doom and gloom
And I'm sat with three screens around me in my room

But there's people to chat to, and faces we know
And new things to learn and we're all on the go
Algo en espanol - there's no cause for alarm
An Indic track showcasing Malayalam

I'm sat in my room, it's all doom and gloom
And I'm sat with my Thinkpads and cats in my room

With webcams and buffering, with lag and delay
It's as well that there's Debconf time all through the day
The effects of tiredness are hard to foresee
For the Debian clocks all are timezone UTC

I'm sat in my room, it's all doom and gloom
And I'll sing out of tune as I'm sat in my room

There's no social drinking, there's no games of Mao
Keeping social distance, we can't think quite how
This year is still friendly though minus some fun
We'll catch up next year when we'll all get some sun

I'm sat in my room, it's all doom and gloom
I'm sat with my friends around here in my room

There's loopy@debconf and snippets and such
To cheer us all up, sure, it doesn't take much
For we're all one big family, though we each code alone
And we sometimes switch off or just complain and moan

I'm sat in my room, it's all doom and gloom
And there's space for us all in the debconf chat room

This is my first DebConf - hope it won't be my last
And we'll meet up somewhere when this COVID is past
To all who have done this - we deserve the credit
Now if you'll excuse me - I've web pages to edit

I'm sat in my room, it's not all doom and gloom
And we're met as one Debian here in my room

The nanotime 0.3.0 release four days ago was so exciting that we decided to do it again! Kidding aside, and fairly extensive tests notwithstanding we were bitten by a few build errors: who knew clang on macOS needed extra curlies to be happy, another manifestation of Solaris having no idea what a timezone setting America/New_York is, plus some extra pickyness from the SAN tests and whatnot. So Leonardo and I gave it some extra care over the weekend, uploaded it late yesterday and here we are with 0.3.1. Thanks again to CRAN for prompt processing even though they are clearly deluged shortly before their (brief) summer break. nanotime relies on the RcppCCTZ package for (efficient) high(er) resolution time parsing and formatting up to nanosecond resolution, and the bit64 package for the actual integer64 arithmetic. Initially implemented using the S3 system, it has benefitted greatly from work by Leonardo Silvestri who rejigged internals in S4 and now added new types for periods, intervals and durations. The NEWS snippet adds full details.

Changes in version 0.3.1 (2020-08-09)

• Several small cleanups to ensure a more robust compilation (Leonardo and Dirk in #75 fixing #74).
• Show Solaris some extra love by skipping tests and examples with a timezone (Dirk in #76).

Thanks to CRANberries there is also a diff to the previous version. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.